Apple Files Patent for Potential New Dimension to Shazam for Wide Range of Devices, Including MR Headset
(Click on the image to greatly enlarge it)
On Thursday, the US Patent and Trademark Office published a patent application from Apple for a possible next-generation feature for their song identification app.Shazam.’ The patent suggests that a next generation of this app will work on many other devices (headset, iPhone, mixed reality headset, iPad, smart contact lenses, head-up display on a vehicle’s windshield, etc.) .
More importantly, the patent describes an all-new feature that could determining that a user is interested in audio content by determining that a movement, such as a nodand trigger the app to identify the melody you enjoy based on the movement of your head to the beat.
The method identifies a time-based relationship between one or more elements of the audio and one or more aspects of bodily motion based on the first sensor data and the second sensor data.
For example, this may involve determining that a device user is moving their head in time to music that is playing aloud in the physical environment. Such head movement can be recognized as a passive indication of interest in music.
In another example, the user’s movement is recognized as an indication of interest based on its type (eg, corresponding to excited behavior) and/or the next movement shortly after a significant event occurs. For instance, this may involve determining that a particular song is playing and the user is interested in the song based on their movement matching the beat of the song.
Various actions can be proactively performed based on the identification of an interest in the content. For example, the device may present identification of the content (eg, display name of song, artist, etc.), present text corresponding to the words of the content (eg, lyrics) and/or present a selectable option to replay the content, continue to experience the content after leaving the physical environment, purchase the content, download the content, and/or add the content to a playlist.
In another example, a characteristic of the content (eg, type of music, tempo range, type(s) of instruments, emotional mood, category, etc.) is identified and used to identify additional content for the user. .
Device resources can be effectively used to determine that a user is interested in audio content. This may involve going through different power states based on different device-level triggers. For example, audio analysis can be done selectively, e.g., based on the detection of body movement, for example, a nod, a tap of the foot, a jump of joy, a first pump, a facial reaction or any other movement indicating the interest of the user.
Apple’s patent FIG. 3 below illustrates the example of electronic device of FIG. 1 obtain motion data according to the disclosed implementations; ]FIG. 4 is a flowchart illustrating a method for identifying interest in audio content by determining that motion has a time-based relationship to detected audio content.
(Click on image to enlarge)
Apple’s patent FIG. 2 below is a view of a mixed reality headset in which the user can see augmented content (#265) that includes a tooltip with information and features selected based on the detection of a body movement and audio in the physical environment.
(Click on image to enlarge)
In some implementations, electronic device #105 of FIG. 2 above generates and presents an extended reality (XR) environment to one or more users. An extended reality (XR) environment refers to a fully or partially simulated environment that people sense and/or interact with through an electronic device.
For details, see Apple patent application number 20220291743.
Brian Temple: Software Engineer, Technology Development Group
Devin Chalmers: Experience Prototyping Lead
Tom Salter: Senior Director of Engineering