Tensorflow JS - Have a dance battle remotely

[lpmi-13]

apparently Ben tells me that you can use tensorflow IN THE BROWSER (!!!) combined with a webcam to get data about body positions, and then somehow use this to either generate an avatar, or in some other way create raw material for a dance battle.

I will attempt to experiment with this at some point.

[benfoxbotica]

• Article https://blog.tensorflow.org/2018/05/real-time-human-pose-estimation-in.html
• Demo https://storage.googleapis.com/tfjs-models/demos/posenet/camera.html

[lpmi-13]

I'm now thinking of applications where you could judge, all in the browser of course, how closely somebody is actually following the beat. I'm assuming the audio stream might have some sort of beat infomation in there...

[lpmi-13]

this looks like a good article for beat detection, and I suppose if we know the songs in advance, then we could even preprocess the data so that whole step doesn't need to happen in the browser. So all we need to do is calculate where the beats should be at any given time, and expect some sort of accelerated motion to terminate then maybe? Need to test out and see what kind of data comes back from the pose detection (x,y...vector...etc).

http://joesul.li/van/beat-detection-using-web-audio/

[lpmi-13]

though now I'm thinking it might be easier (and more useful anyway) to do something where we just ask users to move their hands in response to prompts (either written or spoken) and then give feedback about whether they did it correctly.

For example, "move your left hand up", "keep your left hand up", "move both hands down", and so on. It would be a very simple way to practice responding to written/audio input for learners of English (or any language, really, as long as we can get audio samples, or generate them). Basically an automated Simon Says.

For the actual activity, if it's on a phone (which makes it infinitely more accessible), one person holds it and the second person responds to the commands, receiving immediate feedback.

...yes, I like this idea much more, actually

[lpmi-13]

also plonking this here just in case I eventually get around to the "judge if you're dancing to a beat" thing

https://github.com/chrvadala/music-beat-detector

[benfoxall]

@omgmog and I are going to get a repo started 💪

[benfoxall]

https://github.com/remotehack/poses

[omgmog]

I feel like we got the Tensorflow side of this done, but completely shirked the dance battle side. Next time!