princeton-nlp/SWE-bench_Verified
Viewer
ā¢
Updated
ā¢
500
ā¢
47.2k
ā¢
124
Nikita Davidchuk
Ar4ikov
AI & ML interests
nlp, web, opensource, transformers, asr, ser, tts, cv
Recent Activity
liked
a dataset
3 days ago
princeton-nlp/SWE-bench_Verified
liked
a model
6 days ago
ai-sage/Giga-Embeddings-instruct
liked
a model
6 days ago
ai-sage/GigaChat-20B-A3B-instruct-int8
Organizations
Ar4ikov's activity
liked
a
dataset
3 days ago
liked
2
models
6 days ago
liked
a
model
11 days ago
liked
2
Spaces
13 days ago
liked
a
Space
14 days ago
liked
a
model
18 days ago
liked
a
model
19 days ago
upvoted
a
collection
19 days ago
Adding `safetensors` variant of this model
#1 opened about 1 month ago
by
SFconvertbot
Adding `safetensors` variant of this model
#1 opened about 1 month ago
by
SFconvertbot
Adding `safetensors` variant of this model
#2 opened about 1 month ago
by
SFconvertbot
liked
a
model
19 days ago
liked
a
Space
19 days ago
reacted to
m-ric's
post with ā¤ļø
20 days ago
Post
1467
š¦šµš¼ššØš: š® ššŗš®š¹š¹ š²š»š±-šš¼-š²š»š± š®š“š²š»š ššµš®š š°š®š» š»š®šš¶š“š®šš² š®š»š šØš š®š»š± š¼ššš½š²šæš³š¼šæšŗš šŗšš°šµ šÆš¶š“š“š²šæ ššššš²šŗš! š²
A team from NUS and Microsoft just released an agent that can act on any UI (Desktop, Android, Web) without needing additional text information. It works extremely well : they applied their method on a tiny Qwen2-VL-2B, and they managed to beat methods that use either much more powerful vision models (like GPT-4V) without using any additional info (e.g. leveraging the DOM of a webpage) like previous methods did ! šš
They started from the idea that most existing methods rely heavily on text, which makes them less generalizable, while letting aside rich UI structure that user actually rely on when navigating this interfaces.
āļø They put several good ideas to work:
š” Simplify screenshots to the max:
They prune a lot the heavy visual content of UI screenshots, by removing cloned image patches (like any vast patch of the same color will be reduced to a small patch, while maintaining positional embeddings), then group patches from the same GUI elements together to simplify even further
š” Build a truly generalist dataset:
To train a general UI agent, you need trajectories from each possible UI, and express them in a common language. Authors merge datasets like OmniAct for Desktop, Mind2Web for websites, AMEX for Android trajectories to create a high-quality and diverse dataset.
ā”ļø Nice results ensued:
They fine-tune a tiny Qwen-2-VL-2B on their method, and it reaches SOTA on several task (element identification, web navigation), even beating methods that either use additional info from the DOM or use much bigger VLMS like GPT-4v! š
And performance could certainly jump with a slightly bigger vision model. Let's hope the community builds this soon! š
Paper added to my "Agents" collection š m-ric/agents-65ba776fbd9e29f771c07d4e
A team from NUS and Microsoft just released an agent that can act on any UI (Desktop, Android, Web) without needing additional text information. It works extremely well : they applied their method on a tiny Qwen2-VL-2B, and they managed to beat methods that use either much more powerful vision models (like GPT-4V) without using any additional info (e.g. leveraging the DOM of a webpage) like previous methods did ! šš
They started from the idea that most existing methods rely heavily on text, which makes them less generalizable, while letting aside rich UI structure that user actually rely on when navigating this interfaces.
āļø They put several good ideas to work:
š” Simplify screenshots to the max:
They prune a lot the heavy visual content of UI screenshots, by removing cloned image patches (like any vast patch of the same color will be reduced to a small patch, while maintaining positional embeddings), then group patches from the same GUI elements together to simplify even further
š” Build a truly generalist dataset:
To train a general UI agent, you need trajectories from each possible UI, and express them in a common language. Authors merge datasets like OmniAct for Desktop, Mind2Web for websites, AMEX for Android trajectories to create a high-quality and diverse dataset.
ā”ļø Nice results ensued:
They fine-tune a tiny Qwen-2-VL-2B on their method, and it reaches SOTA on several task (element identification, web navigation), even beating methods that either use additional info from the DOM or use much bigger VLMS like GPT-4v! š
And performance could certainly jump with a slightly bigger vision model. Let's hope the community builds this soon! š
Paper added to my "Agents" collection š m-ric/agents-65ba776fbd9e29f771c07d4e
updated
a
model
24 days ago
liked
2
models
26 days ago
liked
a
model
about 1 month ago
