A technology will be developed that uses Apple Vision Pro to operate a robot with gestures to open doorknobs and pick up objects.
Using Apple's headset-type device '
VisionProTeleop/assets/short_paper_new.pdf at main · Improbable-AI/VisionProTeleop · GitHub
https://github.com/Improbable-AI/VisionProTeleop/blob/main/assets/short_paper_new.pdf
???? Want to use your new Apple Vision Pro to control your robot? Want to record how you navigate / manipulate the world to train a policy?
— Younghyo Park (@younghyo_park) March 9, 2024
I developed an app for VisionOS that can stream your head / wrist / finger movements over WiFi, which you can subscribe on any machines using… pic.twitter.com/l5CbeINWWf
Watch scientists control a robot with their hands while wearing the Apple Vision Pro | Live Science
https://www.livescience.com/technology/virtual-reality/watch-scientists-control-a-robot-with-their-hands-while-wearing-the-apple-vision-pro
Apple Vision Pro is a headset that uses full-color pass-through to realize 'spatial computing' that fuses surrounding reality and virtual space, allowing you to create a virtual work environment that is superimposed on real space. People have already been seen walking around town wearing Apple Vision Pro .
Yong-hyo Park , a robotics engineer enrolled in a doctoral course at MIT, and his colleagues have developed an Apple Vision Pro app that tracks human head and hand movements and streams the movements to robots connected to the same Wi-Fi. We have developed ' Tracking Streamer '. You can see how a person wearing Apple Vision Pro moves a quadruped robot using Tracking Streamer in the embedded video below.
[3/6] Look how @gabe_mrgl used this app/library to control his robot! pic.twitter.com/FIQHE7WGxO
— Younghyo Park (@younghyo_park) March 9, 2024
Gabe Margolis, a graduate student at MIT, is wearing Apple Vision Pro.
The arm of the quadrupedal robot moves in conjunction with Mr. Margolis's hand movements.
If you look closely, you can see that when Mr. Margolis opens his hand, the tip of the robot arm opens, and when he closes his hand, the tip closes.
When Mr. Margolis crouched, the robot crouched as well.
If you operate it well, you can even grab the doorknob with the robot arm.
The complex action of ``grabbing the doorknob with the arm, turning it, and then pushing the doorknob to open the door'' was able to be performed intuitively by moving the hand.
A quadrupedal robot was able to enter the room using the Tracking Streamer.
In addition, by operating the robot arm while the quadrupedal robot is crouched, it is also possible to pick up objects that have fallen on the floor.
Tracking Streamer uses ARKit , which superimposes images on the real world using Apple Vision Pro's cameras and sensors, to track hands and heads.
[4/6] The repository also contains visualization codes. You can check how the axes are defined: pic.twitter.com/4S9kym2i5O
— Younghyo Park (@younghyo_park) March 9, 2024
Related Posts:
