Temporal Preference Optimization for Long-Form Video Understanding
Paper
โข
2501.13919
โข
Published
โข
21
Stanford AI
university
AI & ML interests
None defined yet.
Recent Activity
Stanford's activity
nicholswangย
authored
a
paper
10 days ago
nicholswangย
authored
a
paper
18 days ago
nicholswangย
authored
10
papers
about 2 months ago
Action Sensitivity Learning for Temporal Action Localization
Paper
โข
2305.15701
โข
Published
Whitening-based Contrastive Learning of Sentence Embeddings
Paper
โข
2305.17746
โข
Published
Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models
Paper
โข
2305.18010
โข
Published
Describing Differences in Image Sets with Natural Language
Paper
โข
2312.02974
โข
Published
โข
14
Clustering based Point Cloud Representation Learning for 3D Analysis
Paper
โข
2307.14605
โข
Published
JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human Mesh Recovery
Paper
โข
2307.16377
โข
Published
Bird's-Eye-View Scene Graph for Vision-Language Navigation
Paper
โข
2308.04758
โข
Published
VideoAgent: Long-form Video Understanding with Large Language Model as Agent
Paper
โข
2403.10517
โข
Published
โข
33
Why are Visually-Grounded Language Models Bad at Image Classification?
Paper
โข
2405.18415
โข
Published
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Paper
โข
2412.10360
โข
Published
โข
139
Post
553
๐ The Stanford Institute for Human-Centered AI (https://aiindex.stanford.edu/vibrancy/) has released its 2024 Global AI Vibrancy Tool, a way to explore and compare AI progress across 36 countries.
๐ It measures progress across the 8 broad pillars of R&D, Responsible AI, Economy, Education, Diversity, Policy and Governance, Public Opinion and Infrastructure. (Each of these pillars have a number of Sub Indices)
๐ As a whole it is not surprising that the USA was at the top in terms of overall score as of 2023 (AI investment activity is a large part of the economic pillar for example and that is a large part of the overall USA ranking) but drilling in to more STRATEGIC Macro pillars like Education, Infrastructure or R&D reveal interesting growth patterns in Asia (particularly China) and Western Europe that I suspect the 2024 metrics will bear out.
๐ค Hopefully the 2024 Global Vibrancy ranking will break out AI and ML verticals like Computer Vision or NLP and or the AI Agent space as that may also from a global macro level give indications of what is to come globally for AI in 2025.
๐ It measures progress across the 8 broad pillars of R&D, Responsible AI, Economy, Education, Diversity, Policy and Governance, Public Opinion and Infrastructure. (Each of these pillars have a number of Sub Indices)
๐ As a whole it is not surprising that the USA was at the top in terms of overall score as of 2023 (AI investment activity is a large part of the economic pillar for example and that is a large part of the overall USA ranking) but drilling in to more STRATEGIC Macro pillars like Education, Infrastructure or R&D reveal interesting growth patterns in Asia (particularly China) and Western Europe that I suspect the 2024 metrics will bear out.
๐ค Hopefully the 2024 Global Vibrancy ranking will break out AI and ML verticals like Computer Vision or NLP and or the AI Agent space as that may also from a global macro level give indications of what is to come globally for AI in 2025.
Post
806
๐ค๐ป Function Calling is a key component of Agent workflows. To call functions, an LLM needs a way to interact with other systems and run code. This usually means connecting it to a runtime environment that can handle function calls, data, and security.
Per the Berkeley Function-Calling Leaderboard there are only 2 fully open source models (The other 2 in the top 20 that are not closed source have cc-by-nc-4.0 licenses) out of the top 20 models that currently have function calling built in as of 17 Nov 2024.
https://gorilla.cs.berkeley.edu/leaderboard.html
The 2 Open Source Models out of the top 20 that currently support function calling are:
meetkai/functionary-medium-v3.1
Team-ACE/ToolACE-8B
This is a both a huge disadvantage AND an opportunity for the Open Source community as Enterprises, Small Business, Government Agencies etc. quickly adopt Agents and Agent workflows over the next few months. Open Source will have a lot of catching up to do as Enterprises will be hesitant to switch from the closed source models that they may initially build their Agent workflows on in the next few months to an open source alternative later.
Hopefully more open source models will support function calling in the near future.
Per the Berkeley Function-Calling Leaderboard there are only 2 fully open source models (The other 2 in the top 20 that are not closed source have cc-by-nc-4.0 licenses) out of the top 20 models that currently have function calling built in as of 17 Nov 2024.
https://gorilla.cs.berkeley.edu/leaderboard.html
The 2 Open Source Models out of the top 20 that currently support function calling are:
meetkai/functionary-medium-v3.1
Team-ACE/ToolACE-8B
This is a both a huge disadvantage AND an opportunity for the Open Source community as Enterprises, Small Business, Government Agencies etc. quickly adopt Agents and Agent workflows over the next few months. Open Source will have a lot of catching up to do as Enterprises will be hesitant to switch from the closed source models that they may initially build their Agent workflows on in the next few months to an open source alternative later.
Hopefully more open source models will support function calling in the near future.
leamhadzicย
authored
a
paper
2 months ago
Post
2277
The Mystery Bot ๐ต๏ธโโ๏ธ saga I posted about from earlier this week has been solved...๐ค
Cohere for AI has just announced its open source Aya Expanse multilingual model. The Initial release supports 23 languages with more on the way soon.๐ ๐
You can also try Aya Expanse via SMS on your mobile phone using the global WhatsApp number or one of the initial set of country specific numbers listed below.โฌ๏ธ
๐WhatsApp - +14313028498
Germany - (+49) 1771786365
USA โ +18332746219
United Kingdom โ (+44) 7418373332
Canada โ (+1) 2044107115
Netherlands โ (+31) 97006520757
Brazil โ (+55) 11950110169
Portugal โ (+351) 923249773
Italy โ (+39) 3399950813
Poland - (+48) 459050281
Cohere for AI has just announced its open source Aya Expanse multilingual model. The Initial release supports 23 languages with more on the way soon.๐ ๐
You can also try Aya Expanse via SMS on your mobile phone using the global WhatsApp number or one of the initial set of country specific numbers listed below.โฌ๏ธ
๐WhatsApp - +14313028498
Germany - (+49) 1771786365
USA โ +18332746219
United Kingdom โ (+44) 7418373332
Canada โ (+1) 2044107115
Netherlands โ (+31) 97006520757
Brazil โ (+55) 11950110169
Portugal โ (+351) 923249773
Italy โ (+39) 3399950813
Poland - (+48) 459050281
- 1 reply
Post
2520
Spent the weekend testing out some prompts with ๐ต๏ธโโ๏ธMystery Bot๐ต๏ธโโ๏ธ on my mobile... exciting things are coming soon for the following languages:
๐Arabic, Chinese, Czech, Dutch, English French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese!๐
๐Arabic, Chinese, Czech, Dutch, English French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese!๐
Post
1391
๐ข 2024 CVPR Videos Are Now Available! ๐ฅ
CVPR conference keynotes, panels, posters, workshops, and other content are now available.
โฌ๏ธ
https://cvpr.thecvf.com/Conferences/2024/Videos
CVPR conference keynotes, panels, posters, workshops, and other content are now available.
โฌ๏ธ
https://cvpr.thecvf.com/Conferences/2024/Videos
Post
2351
๐กAndrew Ng recently gave a strong defense of Open Source AI models and the need to slow down legislative efforts in the US and the EU to restrict innovation in Open Source AI at Stanford GSB.
๐ฅSee video below
https://youtu.be/yzUdmwlh1sQ?si=bZc690p8iubolXm_
๐ฅSee video below
https://youtu.be/yzUdmwlh1sQ?si=bZc690p8iubolXm_
- 4 replies
Post
810
Researchers from Auburn University and the University of Alberta have explored the limitations of Vision Language Models (VLMs) in their recently published paper titled "Vision language models are blind." (
Vision language models are blind (2407.06581))
Key Findings:๐
VLMs, including GPT-4o, Gemini-1.5 Pro, Claude-3 Sonnet, and Claude-3.5 Sonnet, struggle with basic visual tasks.
Tasks such as identifying where lines intersect or counting basic shapes are challenging for these models.
The authors noted, "The shockingly poor performance of four state-of-the-art VLMs suggests their vision is, at best, like of a person with myopia seeing fine details as blurry, and at worst, like an intelligent person that is blind making educated guesses"โ(Vision Language Models Are Blind; 2024)โ.
Human-like Myopia? ๐
VLMs may have a blind spot similar to human myopia.
This limitation makes it difficult for VLMs to perceive details.
Suggests a potential parallel between human and machine vision limitations.
Technical Details: ๐ง
The researchers created a new benchmark called BlindTest.
BlindTest consists of simple visual tasks to evaluate VLMs low-level vision capabilities.
Four VLMs were assessed using BlindTest.
Many shortcomings were revealed in the models ability to process basic visual information.
Learn More: ๐ผ๏ธ
For a deeper dive into this research, check out the project page: https://vlmsareblind.github.io/
Key Findings:๐
VLMs, including GPT-4o, Gemini-1.5 Pro, Claude-3 Sonnet, and Claude-3.5 Sonnet, struggle with basic visual tasks.
Tasks such as identifying where lines intersect or counting basic shapes are challenging for these models.
The authors noted, "The shockingly poor performance of four state-of-the-art VLMs suggests their vision is, at best, like of a person with myopia seeing fine details as blurry, and at worst, like an intelligent person that is blind making educated guesses"โ(Vision Language Models Are Blind; 2024)โ.
Human-like Myopia? ๐
VLMs may have a blind spot similar to human myopia.
This limitation makes it difficult for VLMs to perceive details.
Suggests a potential parallel between human and machine vision limitations.
Technical Details: ๐ง
The researchers created a new benchmark called BlindTest.
BlindTest consists of simple visual tasks to evaluate VLMs low-level vision capabilities.
Four VLMs were assessed using BlindTest.
Many shortcomings were revealed in the models ability to process basic visual information.
Learn More: ๐ผ๏ธ
For a deeper dive into this research, check out the project page: https://vlmsareblind.github.io/