https://arxiv.org/api/tQ7AVWKSGtthCVYcDgV6dSmJpJE 2026-09-16T17:45:06Z 20 467 0 http://arxiv.org/abs/2609.15067v1 2026-09-14T05:32:14Z

Procedural audio has emerged as a viable source for transferable audio representation learning, but its design principles remain unclear.We revisit two questions: how a procedural source should be scaled, and whether training choices developed on natural audio should transfer unchanged to procedural data.Using a controlled source, we separate scale into formula-class coverage C and within-class rendering diversity I.Experiments with FDSL and AudioMAE show that these two forms of scale provide different benefits and depend on the learning formulation and downstream task. A matched AudioMAE study further shows that procedural audio favors low mask ratios (10%--25%), whereas AudioSet-28K favors 50%--75%. Shared-codebook analysis reveals lower patch diversity and stronger temporal predictability in procedural audio. These results motivate source-aware procedural pre-training, where source scaling and learning configuration are considered jointly.Code is available at https://github.com/Cross-Innovation-Lab/Formula-Bank.

2026-09-14T05:32:14Z Submitted to ICASSP2027 Jiajun Peng Fengrui Liu Xinyu Liu Feng Liu http://arxiv.org/abs/2609.09368v1 2026-09-08T19:01:22Z

We present the Living Library, an end-to-end framework for transforming fragmented digital archives into governed, conversational, in-person exhibit experiences. Developed and deployed at the Theodore Roosevelt Presidential Library, the framework comprises four layers: digitization and corpus creation, AI-powered processing, retrieval and reasoning, and an optional embodied conversational interface. The first three layers aggregate a 300,000-record collection, apply OCR and structured metadata enrichment for expert curatorial review, and publish records to a hybrid dense/semantic index. Expert review is conducted through the Archivist App, a curator-facing interface that supports correction of AI-generated transcriptions and metadata. The governed corpus powers both a researcher-facing interface and Talk to TR, a continuously operating exhibit that embodies Theodore Roosevelt as a full-scale digital human within a museum environment. To support live, face-to-face interactions, Cross-Era Analogical Grounding reframes contemporary questions through documented historical parallels, allowing Roosevelt to address present-day topics without inventing facts. Dual-path retrieval and end-to-end streaming keep responses grounded and responsive. Layered watchdogs, visitor-session isolation, automated conversation management, and independently restartable services enable reliable unattended operation for hundreds of visitors. Avatar realism, spatial audio, lighting, staging, and conversational design are developed and evaluated as an integrated experience. Rather than report a controlled benchmark, we describe lessons from operating Talk to TR as a public exhibit and offer a transferable model for transforming archival collections into believable, in-person conversational experiences.

2026-09-08T19:01:22Z 25 pages, 6 figures, 4 tables Pengce Wang Lucia Ronchi Darre Matt Briney Michaell Bakalars Dan Rutkowski Ursula Hardy David Wolf Laura Hoffman Allen Kim Shawn Wright Juan Lavista Ferres http://arxiv.org/abs/2609.08542v1 2026-09-08T10:26:01Z

Immersive virtual listening relies on spatial audio technologies such as Higher-Order Ambisonics (HOA), which represent sound scenes as multichannel signals. As the desired spatial resolution increases, so does the number of channels, making efficient compression essential for transmission over bandwidth-limited networks. Moreover, to facilitate deployment by network operators, the target bitrate for immersive audio coding should ideally remain close to the 25 kbps currently allocated to VoLTE audio services. State-of-the-art parametric codecs, such as the recently standardized Immersive Voice and Audio Services (IVAS) codec, achieve compression by transmitting spatial metadata together with a reduced number of transport channels. However, recent studies have shown that IVAS performance degrades on reverberant content, particularly at low bitrates, a limitation that suggests its inability to accurately model room acoustics. In this paper, we propose a novel HOA coding scheme based on the explicit and blind estimation of the Relative Spatial Room Impulse Response (ReSRIR), using a beamformed version of the HOA signal as a reference signal. By exploiting the structure and sparsity of the estimated ReSRIR, we derive an efficient parametric representation for immersive audio coding. Experimental evaluations show that the proposed method achieves higher compression than IVAS in the single-transport-channel regime, while maintaining comparable to slightly better quality.

2026-09-08T10:26:01Z Submitted to International Networked Immersive Audio 2026 (satellite Event of IEEE IS2 2026) Nour Bouayed Adrien Llave Jérôme Daniel Pascal Scalart http://arxiv.org/abs/2609.06837v1 2026-09-06T21:20:36Z

Embodied artificial intelligence has historically very much relied on visual perception, leading to a proliferation of multiple vision-centric world models. However, this reliance fails to capture spatial understanding in its entirety and can even present vulnerabilities in environments with visual occlusions, low-light conditions, or blackouts-scenarios, where acoustic information becomes a critical alternative for spatial awareness and navigation. Despite its potential, research into realistic spatial audio and particularly the development of audio-centric world models remains sparse. In this technical report, we introduce BinauralVAE: a flexible, open-source pipeline (https://github.com/Luizerko/BinauralVAE) that explores multiple models for spatialized audio reconstruction, progressing from fundamental baselines to advanced, mathematically grounded architectures. Our approach evaluates various Variational Autoencoder architectures -- including complex-valued variants -- to learn robust latent representations of binaural signals. Developed alongside AudioWorldSim, our methodology leverages realistic acoustic data captured as a simulated robot navigates an environment. This pipeline establishes a foundation for state representation in a future audio-based world model, designed to map the direct causal connection between navigational actions and their resulting acoustic consequences, and helping to enable sound as an essential complementary modality for spatial knowledge acquisition.

2026-09-06T21:20:36Z 17 pages, 7 figures Luis Vitor Zerkowski Luiz Velho http://arxiv.org/abs/2609.04975v2 2026-09-08T07:34:46Z

Spatial audio editing modifies an existing soundfield according to a user's instruction while preserving the rest of the scene. Unlike conventional audio editing, it must reason jointly about audio events, spatial information, dynamic changes, and environmental information in first-order Ambisonic (FOA) waveforms. Existing language-guided editors mainly target conventional audio or rely on sequential operations, and therefore do not directly support one-stage editing for complex 3D spatial instructions. We present SwanWeave, the first one-stage multi-task framework for instruction-guided 3D FOA spatial audio editing. We build paired FOA supervision from open-source speech and sound-effect corpora using controllable room simulation, covering more than ten single-operation and compound tasks across the four editing axes. To handle this heterogeneous edit space, SwanWeave uses Spatial Edit Mixture-of-Experts (SE-MoE) with dual-level routing, selecting task-aware expert combinations for compound instructions and frame-level routed/null experts for local edit decisions. We further introduce Spatial Preference Optimization (SPO), a Direct Preference Optimization (DPO)-based alignment objective with edit-specific negative targets, and adopt staged training to improve natural-language grounding. Experiments show that SwanWeave achieves better editing quality than existing general audio editors and spatial audio baselines across all tasks. Spatial audio editing demos can be found at https://swanaigc.github.io/#swanweave, code can be found at: https://github.com/MM-Speech/SwanWeave.

2026-09-04T10:27:52Z Ke Lei Chenyuhao Wen Yu Zhang Wenxiang Guo Changhao Pan Sashuai Zhou Yongshi Li Ruiqi Li Ruofan Hu Haorui Xu Xiang Yin Zhou Zhao http://arxiv.org/abs/2609.00648v1 2026-09-01T03:25:03Z

For noisy real-world environments such as those in open public spaces, spoken dialogue systems for both autonomous robots and avatars should be carefully designed to provide enhanced speech signals. These signals can be used either for speech recognition or, in the case of an avatar system, transmitted as clean speech to a remote operator. This work proposes an audio system that can be used for both these scenarios and was demonstrated as a proof-of-concept at the 2025 World Expo in Osaka. The first scenario is an attentive listening system with the android ERICA, and the second is a conversation support system with mobile Teleco robots, with one of them acting as an avatar for a remote operator. Both systems feature multi-party conversation and use a single multi-channel microphone array. We describe how our audio system not only enhances the speech of multiple speakers in a noisy environment, but provides a form of spatial audio which allows for more immersiveness in avatar-based conversational interactions.

2026-09-01T03:25:03Z Divesh Lala Yogeeswaran Muthukumaran Vincent Fernandes Kazushi Kato Shota Fujiki Zihao Chi Masaya Iwasaki Taiken Shintani Megumi Kawata Kazuki Sakai Koji Inoue Yuicihiro Yoshikawa Tatsuya Kawahara http://arxiv.org/abs/2608.29826v1 2026-08-30T14:48:18Z

We construct temperature-dependent deep potentials for sodium using finite-temperature DFT data obtained with the PBE, AM05, and $r^2$SCAN exchange-correlation functionals and investigate the thermophysical properties of sodium from room temperature to the critical region. The predicted critical parameters show a dependence on the exchange-correlation functional. The model based on the $r^2$SCAN functional gives a critical temperature $T_c=2.508(8)$ kK, density $ρ_c=0.203(4)$ g/cm$^{3}$, and pressure $P_c=0.249(6)$ kbar, in close agreement with the recommended values. This model is then used to reconstruct the normal-pressure and critical isobars and to calculate the enthalpy, heat capacities, thermal expansion coefficient, bulk moduli, Grüneisen parameter, and speed of sound. The calculated normal-pressure bcc density differs by only 0.06% from the experimental value. Direct solid-liquid coexistence simulations give a melting temperature of $346(2)$ K, 25 K below the recommended value, consistent with the sensitivity expected from meV/atom free-energy errors. Liquid-vapor coexistence simulations reproduce the binodal and yield a surface tension that approaches zero near the critical point. The calculated self-diffusion coefficient and shear viscosity extend the available transport-property data into the expanded-liquid and near-critical regions, where direct experimental information is sparse. We also demonstrate the thermodynamic consistency of the results by comparing the speed of sound obtained from direct acoustic simulations with that calculated from the equation of state.

2026-08-30T14:48:18Z 18 pages, 19 figures G. S. Demyanov D. V. Minakov V. B. Fokin P. R. Levashov http://arxiv.org/abs/2608.29549v1 2026-08-30T04:37:50Z

Text-to-spatial audio generation, such as text-to-First-Order Ambisonics (FOA), provides a convenient way to create spatial audio for billion-dollar gaming and film industries. However, existing text-to-FOA methods are largely data-driven and may produce audio that violates acoustic relations between source direction and distance. They also separate descriptive and parametric control, forcing users to trade usability for precision. In this paper, we present PhysWave, a physics-guided latent diffusion model for controllable text-to-FOA generation. PhysWave unifies natural-language and trajectory control through a shared waypoint-caption representation, and augments diffusion training with two differentiable acoustic priors: spherical-harmonic direction consistency and inverse-square distance consistency. To support dynamic spatial generation, we further construct a 300K-clip FOA dataset with diverse sound categories and source trajectories. Extensive results show that the proposed priors help PhysWave generate spatially consistent FOA audio while maintaining competitive audio quality. Further analyses show that these physics priors improve spatial consistency during training and can also be used as inference-time guidance for training-free spatial refinement.

2026-08-30T04:37:50Z Accepted by EMNLP 2026 Main Conference. Project website: https://lingfengyao.github.io/PhysWave/ Lingfeng Yao Chenpei Huang Xingke Yang Ziye Geng Changqing Luo Hao Wang Jiang Liu Miao Pan http://arxiv.org/abs/2608.27648v1 2026-08-27T19:36:38Z

Oscillator synchronization is a widely used sound-synthesis technique, but straightforward digital implementations suffer from aliasing artifacts. This paper presents an alias-free method for digital emulation of oscillator synchronization of arbitrary periodic waveforms based on additive synthesis. Starting from a finite set of Fourier-series coefficients representing a bandlimited free-running waveform, we derive linear spectral-resampling transforms that map these coefficients to those of the bandlimited synchronized waveform. Beyond conventional hard synchronization, the proposed approach also supports two additional soft-synchronization modes. To address the high computational complexity of the proposed method, we introduce HASY, a 6 mm^2 application-specific integrated circuit (ASIC) fabricated in 65 nm CMOS technology. HASY generates one 96 kHz, 24 bit alias-free synchronized waveform with up to 512 harmonics and computes the spectral-resampling transform within only five audio-sample periods.

2026-08-27T19:36:38Z To be presented at DAFx26 Jonas Roth Domenic Keller Oscar Castañeda Christoph Studer http://arxiv.org/abs/2608.26566v1 2026-08-27T03:17:55Z

While contemporary neural text-to-speech (TTS) systems have achieved high levels of intelligibility, they frequently lack the emotional nuance required for authentic affective communication. This limitation is particularly critical in Extended Reality (XR), where the absence of emotionally expressive audio can diminish user presence and spatial immersion. We present EmoSay, an Artificial Intelligence-driven Text-to-Emotional-Speech (TTES) system designed to bridge the semantic-affective gap in immersive environments. EmoSay modulates a neural synthesis pipeline using discrete emotional prompts, delivering the output through a Unity-based interface featuring high-fidelity spatialized audio. The system was evaluated through a comprehensive user study focusing on perception, engagement, and the subjective sense of empathy. Our results demonstrate that EmoSay significantly enhances the immersive experience, achieving a System Usability Scale (SUS) score of 74.76, indicating strong usability and seamless integration within the XR workflow. Subjective assessments reveal a high degree of perceived naturalness and a strong positive correlation between emotional expressiveness and user engagement. Regression analysis identifies vocal naturalness as the strongest of the tested predictors of user satisfaction, suggesting that EmoSay's affective prosody helps meet the heightened expectations for realism in immersive settings. This work contributes a scalable, affect-aware framework for inclusive XR design and demonstrates the role synthetic emotion can play in fostering human-computer rapport through voice-first interaction.

2026-08-27T03:17:55Z Sikiru Ademola Adewale Sunday D. Ubur Nikitha Donekal Chandrashekar Onyeka Emebo Denis Gračanin 10.1007/978-3-032-31421-5_1 http://arxiv.org/abs/2608.25404v1 2026-08-26T06:06:05Z

Spatial audio vocoders are able to convert mel-spectrograms produced by generative models into spatial audio waveforms. Most neural vocoders are designed for monaural audio, and direct extensions to spatial audio can degrade spatial quality by ignoring inter-channel cues. We present CSAVocoder, a causal GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering. Our framework introduces a Spatial Adaptor that fuses multi-channel mel-spectrograms with dynamic source-listener pose information, together with a spatial consistency discriminator that supervises inter-channel cues. To meet real-time requirements, we design a strictly causal, stateful generator that supports efficient streaming inference with constant memory overhead. Experiments on large-scale spatial audio datasets show that CSAVocoder improves spatial fidelity at competitive audio quality and real-time performance.

2026-08-26T06:06:05Z Zhiyuan Zhu Han Wang Wenxiang Guo Yu Zhang Changhao Pan Rui Yang Zhou Zhao http://arxiv.org/abs/2608.24579v1 2026-08-25T14:04:54Z

Spatial audio is a key component of immersive $360^\circ$ media, yet high-quality spatial capture remains limited in real-world speech-dominant scenes. We study visually guided First-Order Ambisonics (FOA) speech spatialization in the wild: given aligned $360^\circ$ video and an omnidirectional audio track, we recover the missing directional FOA components. To support this task, we introduce YT-SPEECH, a speech-oriented $360^\circ$ video-FOA dataset curated from YouTube. We propose a two-stage Localizer-Renderer framework, where an audio-visual segmentation backbone provides frame-wise spatial heatmaps and a conditional complex-domain U-Net reconstructs directional FOA signals from the omnidirectional channel. A confidence-based gating strategy stabilizes conditioning under ambiguous acoustic conditions. Experiments show improved reconstruction fidelity, spatial accuracy, and perceptual speech quality relative to ablated variants and prior approaches.

2026-08-25T14:04:54Z Accepted at INTERSPEECH 2026 Qingyu Luo Peng Zhang Wenwu Wang Philip J. B. Jackson http://arxiv.org/abs/2608.24558v3 2026-09-08T15:48:00Z

Spatial audio enhances user immersion by reproducing 3D sound fields, with Ambisonics being a widely adopted representation. While Ambisonics is theoretically independent of the recording setup, practical microphone arrays introduce hardware-dependent encoding artifacts. Moreover, existing data-driven solutions lack flexibility, as they are typically restricted to fixed array geometries. To overcome these limitations, we propose ADEPS, a generative framework that explicitly embeds the physical acquisition model into the inference process. By leveraging this formulation, ADEPS effectively compensates for array-specific distortions while enabling zero-shot encoding across arbitrary array topologies. We train the underlying generative prior in an unsupervised manner solely on target Ambisonic representations. Extensive evaluations across diverse simulated and real microphone arrays demonstrate that ADEPS consistently outperforms both traditional linear and parametric baselines in spatial fidelity and spectral quality.

2026-08-25T13:43:31Z Amit Milstein Nir Shlezinger Boaz Rafaely http://arxiv.org/abs/2608.24374v1 2026-08-25T10:35:06Z

Conventional audio language models (ALMs) have made significant progress in achieving alignment between auditory and textual representations, including recent explorations in spatial audio. However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events. To overcome these limitations, we propose CoSTALA-a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity. Extensive experiments demonstrate that CoSTALA significantly establish a powerful new framework for spatio-temporal audio understanding.

2026-08-25T10:35:06Z Peiwei Ren Jinbo Hu Fang Kang Shan Liang Yin Cao http://arxiv.org/abs/2608.20971v1 2026-08-21T10:50:00Z

We investigate how the realism of synthetic room impulse response (RIR) datasets affects the training of DeepFilterNet3 for single-channel speech enhancement. We compare a DNS4 image-source-method (ISM) RIR dataset with a higher-acoustic-fidelity dataset generated using hybrid wave-based and geometrical acoustics simulation. Rather than isolating individual simulation factors, we compare complete RIR generation pipelines while keeping the enhancement model unchanged. Models are evaluated on unseen measured RIRs using objective speech enhancement metrics and downstream automatic speech recognition (ASR). Training with the higher-fidelity dataset consistently yields modest improvements in objective metrics and substantially lower ASR word error rates than the ISM dataset. Although the experiments do not attribute these gains to individual modelling components, they show that increasing the overall realism of synthetic acoustic training data improves the generalization of DeepFilterNet3 to unseen measured environments.

2026-08-21T10:50:00Z 5 pages, 2 figures, IWAENC 2026 Alessia Milo Georg Götz Steinar Guðjónsson Daniel Gert Nielsen Jesper Pedersen Finnur Pind http://arxiv.org/abs/2608.16722v2 2026-08-19T12:38:19Z

Individually measuring head-related transfer functions (HRTFs) at scale remains a central challenge for personalised spatial audio, motivating growing interest in synthetic HRTFs. We evaluated the numerical, computational, and behavioural validity of synthetic HRTFs, generated through the boundary element method simulation using Mesh2HRTF, against measured and KEMAR HRTFs using the Extended SONICOM dataset. Across 200 subjects, synthetic HRTFs deviated less from measured than KEMAR in interaural time and level differences, but residual errors, together with elevated spectral distortion, concentrated at low, rear elevations. This is consistent with the omission of torso geometry from the synthesis pipeline. Two computational models revealed a corresponding pattern of predicted localisation errors, with synthetic HRTFs positioned between measured and KEMAR. In a virtual reality localisation task (N = 20), synthetic HRTFs matched measured on every polar metric, while KEMAR was significantly worse. However, behavioural error clustered around the front-back midline regardless of condition, not at the low elevations implicated numerically or by the models. A separate spatial release from masking task (N = 18) showed no effect of HRTF type. Together, these results indicate that high-resolution synthetic HRTFs preserve behavioural localisation performance, despite discrepancies between the numerical/model-predicted bias and the spatial pattern of behavioural error.

2026-08-17T15:38:08Z Version 2 clarifies in the pdf this is a preprint submitted to JASA and awaiting review Katarina C. Poole Lorenzo Picinali http://arxiv.org/abs/2608.16240v1 2026-08-17T08:17:21Z

Ambisonics delivers compact scene based spatial audio representation, yet higher order Ambisonic encoding poses difficulties for wearables and embedded hardware. Their microphone arrays are often sparse, irregular, and constrained by device specific boundary conditions. These factors make the spherical-harmonic (SH) domain encoding ill conditioned: inverse filtering amplifies noise, while deterministic neural encoders may overfit to array-specific responses or smooth ambiguous higher-order components. This paper presents DiffM2A, a geometry-adaptive conditional diffusion framework for robust Ambisonic encoding from sparse MAs with variable topologies. Its Geometry-Adaptive Spherical Harmonic Projection (GASHP) front-end constructs boundary-aware SH steering functions and applies an energy-normalized modal projection, mapping array-dependent observations to a common modal representation without explicit pseudo-inverse computation. A dual-branch Elucidated Diffusion Model then estimates complex Ambisonic coefficients, conditioned on both the raw microphone spectra and GASHP features. Sound intensity and rotational equivariance losses further enhance inter-channel phase consistency and structured behavior across SH subspaces. Evaluations on both first- and second-order Ambisonic encoding tasks, using simulated room-acoustics and real-world LOCATA recordings, demonstrate that DiffM2A outperforms conventional and neural baseline methods on signal fidelity, spectral accuracy, spatial coherence, and binaural cue preservation. Additional experiments show that these gains are largely retained across unseen five-microphone layouts and under mismatched open-array and rigid-sphere boundary models.

2026-08-17T08:17:21Z Xiang Zhou Zhengqiao Zhao Zhengding Luo Wen Zhang http://arxiv.org/abs/2608.14097v2 2026-08-17T14:27:56Z

We address the problem of encoding room impulse responses (RIRs) into high-order Ambisonics (HOA) representations from arbitrary and potentially insufficient or incomplete microphone array measurements. This task is fundamentally ill-posed for microphone arrays with limited spatial capture capabilities, such as irregular or sparse arrays, as classical linear methods fail to reconstruct high-order spatial detail. We introduce a diffusion-based generative framework that models the statistical properties of HOA RIRs. This enables device-agnostic encoding from arbitrary microphone arrays, potentially unseen during data measurement. Our approach incorporates a posterior sampling procedure that enforces consistency between the estimated signals and the measurements while plausibly reconstructing spatial information that is unobservable from the limited measurements alone. Experiments on simulated data demonstrate that our method outperforms linear and neural baselines, achieving accurate HOA RIR estimation up to 12th order. A listening test with binaural renderings, including both simulated and measured RIRs, further confirms that the proposed method yields higher perceptual similarity to reference Ambisonics RIRs than all baselines. The flexibility and accuracy of the proposed framework opens new possibilities for scalable acoustics simulations.

2026-08-14T08:57:32Z IWAENC 2026 Eloi Moliner Christoph Hold Juan Azcarreta Ortiz Sebastian Prepelita Ishwarya Ananthabhotla Daniel Wong Sanjeel Parekh Sanha Lee http://arxiv.org/abs/2608.09435v1 2026-08-10T11:06:28Z

Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83\% average semantic accuracy across the four levels, compared with 37.28\% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.

2026-08-10T11:06:28Z Zhi Zeng Cheng Zhang Zesheng Yang Rendong Pi Jiaying Wu Di Zhang Zihan Ma Guodong Li Zhou Yang Yu Xiang Yifei Zheng Minnan Luo http://arxiv.org/abs/2608.05145v2 2026-08-06T01:45:56Z

While modern 3D reconstruction excels at modeling object geometry and appearance, it largely ignores the rich acoustic cues revealed through physical interaction. Object impact sounds convey material, stiffness, and structural properties that complement vision, yet existing impact sound modeling approaches either rely on expensive physics-based simulation or require large datasets to generalize in a purely data-driven manner. We introduce Audio-Visual Modal Sound Field (AV-MSF), a novel object-level acoustic representation reconstructed from multi-view images and only a few impact sound recordings. AV-MSF builds on 3D Gaussian Splatting integrated with dense 3D visual feature to provide a strong geometry-aware prior, and represents the impact sound field using compact, physically meaningful modal parameters, enabling robust few-shot reconstruction. Experiments on two real-world datasets show that AV-MSF achieves state-of-the-art impact sound rendering, outperforming both physics-based and data-driven baselines. Furthermore, we demonstrate downstream applications enabled by our representation, including contact localization and object sound editing.

2026-08-05T17:59:19Z ECCV 2026, Project page: https://zisenshao.github.io/AV-MSF/ Zisen Shao Zihao Wei Derong Jin Ruohan Gao