File size: 11,220 Bytes
1c36a14 e7b420b 1c36a14 e7b420b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 9c740c5 1c36a14 630fc1b 1c36a14 9c740c5 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 d03a25b 630fc1b 1c36a14 d03a25b 1c36a14 9c740c5 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 1c36a14 630fc1b 9c740c5 e7b420b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 |
---
license: mit
datasets:
- imageomics/KABR
language:
- en
tags:
- biology
- CV
- images
- animals
- zebra
- giraffe
- behavior
- behavior recognition
- annotation
- UAV
- drone
- video
---
# Model Card for X3D-KABR-Kinetics
X3D-KABR-Kinetics is a behavior recognition model for in situ drone videos of zebras and giraffes,
built using X3D model initialized on Kinetics weights.
It is trained on the [KABR dataset](https://huggingface.co./datasets/imageomics/KABR),
which is comprised of 10 hours of aerial video footage of reticulated giraffes
(*Giraffa reticulata*), Plains zebras (*Equus quagga*), and Grevy’s zebras
(*Equus grevyi*) captured using a DJI Mavic 2S drone.
It includes both spatiotemporal (i.e., mini-scenes) and behavior annotations provided by an expert
behavioral ecologist.
## Model Details
### Model Description
- **Developed by:** [Maksim Kholiavchenko, Maksim Kukushkin, Otto Brookes, Jenna Kline, Sam Stevens, Isla Duporge, Alec Sheets,
Reshma R. Babu, Namrata Banerji, Elizabeth Campolongo,
Matthew Thompson, Nina Van Tiel, Jackson Miliko,
Eduardo Bessa Mirmehdi, Thomas Schmid,
Tanya Berger-Wolf, Daniel I. Rubenstein, Tilo Burghardt, Charles V. Stewart]
- **Model type:** [X3D]
- **License:** [MIT]
- **Fine-tuned from model:** [X3D-S, Kinetics]
This model was developed for the benefit of the community as an open-source product, thus we request that any derivative products are also open-source.
### Model Sources
- **Repository:** [Project Repo](https://github.com/Imageomics/kabr-tools)
- **Paper:** [Paper Link](https://openaccess.thecvf.com/content/WACV2024W/CV4Smalls/papers/Kholiavchenko_KABR_In-Situ_Dataset_for_Kenyan_Animal_Behavior_Recognition_From_Drone_WACVW_2024_paper.pdf)
- **Project Page** [Project Page](https://kabrdata.xyz/)
## Uses
X3D-KABR-Kinetics has extensively studied ungulate behavior classification from aerial video.
### Direct Use
Please see the illustrative examples on the [kabr-tools repository](https://github.com/Imageomics/kabr-tools)
for more information on how this model can be used generate time-budgets from aerial video of animals.
### Out-of-Scope Use
This model was trained to detect and classify behavior from drone videos of zebras and giraffes in Kenya. It may not perform well on other species or settings.
<!--
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
<!--
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations.
This model was trained to detect and classify behavior from drone videos of zebras and giraffes in Kenya. It may not perform well on other species or settings.
-->
## How to Get Started with the Model
Please see the illustrative examples on the [kabr-tools repository](https://github.com/Imageomics/kabr-tools)
for more information on how this model can be used generate time-budgets from aerial video of animals.
## Training Details
### Training Data
[KABR Dataset](https://huggingface.co./datasets/imageomics/KABR)
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing
Raw drone videos were pre-processed using CVAT to detect and track each individual animal in each
high-resolution video and link the results into tracklets.
For each tracklet, we create a separate video, called a mini-scene, by extracting a sub-image centered on each
detection in a video frame.
This allows us to compensate for the drone's movement and provides a stable, zoomed-in representation of the animal.
See [project page](https://kabrdata.xyz/) and the [paper](https://openaccess.thecvf.com/content/WACV2024W/CV4Smalls/papers/Kholiavchenko_KABR_In-Situ_Dataset_for_Kenyan_Animal_Behavior_Recognition_From_Drone_WACVW_2024_paper.pdf) for data preprocessing details.
We applied data augmentation techniques during training, including horizontal flipping to randomly
mirror the input frames horizontally and color augmentations to randomly modify the
brightness, contrast, and saturation of the input frames.
#### Training Hyperparameters
The model was trained for 120 epochs, using a batch size of 5.
We used the EQL loss function to address the long-tailed class distribution and SGD optimizer with a learning rate of 1e5.
We used a sample rate of 16x5, and random weight initialization.
<!-- ADD RESULTS ONCE NEW PAPER PUBLISHED
<!--
#### Speeds, Sizes, Times
<!-- [optional] This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
<!--
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
<!--
[More Information Needed]
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible, otherwise link to the original source with more info.
Provide a basic overview of the test data and documentation related to any data pre-processing or additional filtering. -->
<!--
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
<!--
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
<!--
[More Information Needed]
### Results
[More Information Needed]
#### Summary
[More Information Needed]
## Model Examination
<!-- [optional] Relevant interpretability work for the model goes here -->
<!--
[More Information Needed]
## Environmental Impact
<!--
It would be great to try to include this.
Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
<!--
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute)
presented in [Lacoste et al. (2019)](https://doi.org/10.48550/arXiv.1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications
[More Information Needed--optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed: hardware requirements]
#### Software
[More Information Needed]
## Citation
<!-- If there is a paper introducing the model, the Bibtex information for that should go in this section.
See notes at top of file about selecting a license.
If you choose CC0: This model is dedicated to the public domain for the benefit of scientific pursuits.
We ask that you cite the model and journal paper using the below citations if you make use of it in your research.
-->
**BibTeX:**
If you use our model in your work, please cite the model and associated paper.
**Model**
```
@software{kabr_x3d_model,
author = {Maksim Kholiavchenko, Maksim Kukushkin, Otto Brookes, Jenna Kline, Sam Stevens, Isla Duporge, Alec Sheets,
Reshma R. Babu, Namrata Banerji, Elizabeth Campolongo,
Matthew Thompson, Nina Van Tiel, Jackson Miliko,
Eduardo Bessa Mirmehdi, Thomas Schmid,
Tanya Berger-Wolf, Daniel I. Rubenstein, Tilo Burghardt, Charles V. Stewart},
doi = {<doi once generated>},
title = {KABR model},
version = {v0.1},
year = {2024},
url = {https://huggingface.co./imageomics/x3d-kabr-kinetics}
}
```
**Paper**
```
@InProceedings{Kholiavchenko_2024_WACV,
author = {Kholiavchenko, Maksim and Kline, Jenna and Ramirez, Michelle and Stevens, Sam and Sheets, Alec and Babu, Reshma and Banerji, Namrata and Campolongo, Elizabeth and Thompson, Matthew and Van Tiel, Nina and Miliko, Jackson and Bessa, Eduardo and Duporge, Isla and Berger-Wolf, Tanya and Rubenstein, Daniel and Stewart, Charles},
title = {KABR: In-Situ Dataset for Kenyan Animal Behavior Recognition From Drone Videos},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops},
month = {January},
year = {2024},
pages = {31-40}
}
```
<!-- ADD ONCE PAPER PUBLISHED
@article{kabr2024,
title = {Deep Dive into KABR: A Dataset for Understanding Ungulate Behavior from In-Situ Drone Video},
author = {Maksim Kholiavchenko, Jenna Kline, Maksim Kukushkin, Otto Brookes, Sam Stevens, Isla Duporge, Alec Sheets,
Reshma R. Babu, Namrata Banerji, Elizabeth Campolongo,
Matthew Thompson, Nina Van Tiel, Jackson Miliko,
Eduardo Bessa Mirmehdi, Thomas Schmid,
Tanya Berger-Wolf, Daniel I. Rubenstein, Tilo Burghardt, Charles V. Stewart},
journal = {},
year = 2024,
url = {},
doi = {<DOI>}
}
## Acknowledgements
This work was supported by the [Imageomics Institute](https://imageomics.org),
which is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under
[Award #2118240](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2118240)
(Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning).
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s)
and do not necessarily reflect the views of the National Science Foundation.
<!--
## Glossary
<!-- [optional] If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
<!--
## More Information
<!-- [optional] Any other relevant information that doesn't fit elsewhere. -->
## Model Card Authors
[Jenna Kline and Maksim Kholiavchenko]
## Model Card Contact
Maksim Kholiavchenko
<!-- Could include who to contact with questions, but this is also what the "Discussions" tab is for. -->
### Contributions
This work was supported by the [Imageomics Institute](https://imageomics.org), which is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under [Award #2118240](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2118240) (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Additional support was also provided by the [AI Institute for Intelligent Cyberinfrastructure with Computational Learning in the Environment (ICICLE)](https://icicle.osu.edu/), which is funded by the US National Science Foundation under [Award #2112606](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2112606). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
The data was gathered at the [Mpala Research Centre](https://mpala.org/) in Kenya, in accordance with Research License No. NACOSTI/P/22/18214. The data collection protocol adhered strictly to the guidelines set forth by the Institutional Animal Care and Use Committee under permission No. IACUC 1835F. |