Trying out new technologies with a pun-telling teddy bear

Some time back we had an internal discussion about a potential sales case that sounded particularly interesting. There was no plan or design for this and so I had free rein. A bit before this I had been reading up on computer vision, voice recognition technology and had been playing with ThreeJS, a JavaScript framework that lets us render 3D objects in the browser, so I thought that I would combine these and see what I could come up with.

I was excited to try my hand at it but as it was a potential case, there was no budget for it. To save the day, in comes Exove Labs – an internal project for trying new and cool technologies. The goal of Exove Labs is to create and test prototypes and proof of concepts to bring new opportunities and value to our customers. Finally, I was given the go ahead to see what I could make.

Readymade libraries can get you a long way

I got my hands on a silly 3D model of a teddy bear from friends in Exove Design and I was off. First step was to find a face tracking library. I settled on a simple plug and play library which was enough for my purposes but sadly limited my control greatly when compared to tracking.js or openCV. Tracking.js is a powerful and versatile library for computer vision in the browser and is impressively light weight. OpenCV is an incredible open source computer vision library that can be compiled in JavaScript. It has the most customisation but is also not made with JavaScript in mind and is the heaviest option.

With the plug and play library I was able to detect a person’s face and, with some trigonometry to rotate the model’s head as if it were sitting further into the screen. Hence, we had a model that would look at a person sitting in front of it. I had tried to work with projected depth to get a more accurate reaction but sadly the library that I was using made that all but impossible. The trigonometry, however, provided a perfectly workable solution due to the fact that the model wasn’t trying to focus on the user. It was a teddy bear after all.

Next was to add a bit more magic, I wanted the bear to have some idle animations and some other trigger based animations too. Thankfully ThreeJS made this quite easy. I needed to have the teddy bear’s head separated from its body to have both the animations and head tracking at the same time, but he didn’t seem to mind that too much.

Why not have a little fun?

Now this was truly an adorable little demo but you know what makes everything better? Puns and dad jokes! Making use of the Web Speech API and its implementation in Google Chrome I was able to have the teddy bear “hear” what I was saying thanks to the SpeechRecognition part of the API. With the SpeechSynthesis part of the API I was able to have the teddy speak to me, of course I needed to make its voice super high pitch. Finally, with the help of some various websites I was able to amass a large number of just awful jokes. “All” it took to put it together was some regex, but, after dealing with that problem, my adorable, horribly annoyingly voiced, hilarious demo was complete!

This demo actually helped lead to some proper client work that turned out to be the best ~6 months of my working life to date. One of the most interesting and challenging projects that I have had the pleasure to work on. I got to work with some amazing people and in general it makes me super thankful that I was able to experiment and assist sales in winning the project.

Thoughts by