bpiyush/sound-of-water-models

🚰 The Sound of Water: Inferring Physical Properties from Pouring Liquids

In this folder, we provide the following trained model checkpoints:

Key insight: As water is poured, the fundamental frequency that we hear changes predictably over time as a function of physical properties (e.g., container dimensions).

TL;DR: We present a method to infer physical properties of liquids from just the sound of pouring. We show in theory how pitch can be used to derive various physical properties such as container height, flow rate, etc. Then, we train a pitch detection network (wav2vec2) using simulated and real data. The resulting model can predict the physical properties of pouring liquids with high accuracy. The latent representations learned also encode information about liquid mass and container shape.

Arxiv link: https://arxiv.org/abs/2411.11222

Demo

Check out the demo here. You can upload a video of pouring and the model estimates pitch and physical properties.

💻 Usage

First, install the repository from github.

git clone [email protected]:bpiyush/SoundOfWater.git
cd SoundOfWater

Then, install dependencies.

conda create -n sow python=3.8
conda activate sow

# Install desired torch version
# NOTE: change the version if you are using a different CUDA version
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121

# Additional packages
pip install lightning==2.1.2
pip install timm==0.9.10
pip install pandas
pip install decord==0.6.0
pip install librosa==0.10.1
pip install einops==0.7.0
pip install ipywidgets jupyterlab seaborn

# if you find a package is missing, please install it with pip

Then, use this snippet to download the models:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="bpiyush/sound-of-water-models",
    local_dir="/path/to/download/",
)

To run our models on examples of pouring sounds, please see the playground notebook.

If you would like to use our dataset for a different task, please download it from here.

Models

We provide audio models trained to detect pitch in the sound of pouring water. We train these models in two stages:

1. Pre-training on synthetic data: We simulate sounds of pouring water using DDSP using only 80 samples. This is used to generate lots of simulated sounds of pouring water. Then, we train wav2vec2 on this data.
2. Fine-tuning on real data: We fine-tune the model on real data. Since real data does not come with ground truth, we use visual co-supervision from the video stream to fine-tune the audio model.

Here, we provide checkpoints for both the stages.

File name	Description	Size
dsr9mf13_ep100_step12423_synthetic_pretrained.pth	Pre-trained on synthetic data	361M
dsr9mf13_ep100_step12423_real_finetuned_with_cosupervision.pth	Trained with visual co-supervision	361M

📜 Citation

If you find this repository useful, please consider giving a star ⭐ and citation

@article{sound_of_water_bagad,
  title={The Sound of Water: Inferring Physical Properties from Pouring Liquids},
  author={Bagad, Piyush and Tapaswi, Makarand and Snoek, Cees G. M. and Zisserman, Andrew},
  journal={arXiv},
  year={2024}
}

🙏 Acknowledgements

• We thank Ashish Thandavan for support with infrastructure and Sindhu Hegde, Ragav Sachdeva, Jaesung Huh, Vladimir Iashin, Prajwal KR, and Aditya Singh for useful discussions.
• This research is funded by EPSRC Programme Grant VisualAI EP/T028572/1, and a Royal Society Research Professorship RP / R1 / 191132.

We also want to highlight closely related work that could be of interest:

• Analyzing Liquid Pouring Sequences via Audio-Visual Neural Networks. IROS (2019).
• Human sensitivity to acoustic information from vessel filling. Journal of Experimental Psychology (2020).
• See the Glass Half Full: Reasoning About Liquid Containers, Their Volume and Content. ICCV (2017).
• CREPE: A Convolutional Representation for Pitch Estimation. ICASSP (2018).

🙅🏻 Potential Biases

Our model is based on wav2vec2 which is trained on a large-scale speech recognition data. While this data is not as large as usual datasets in AI, it may still have undesirable biases that are present in the training data.