This DIY Apple Pencil writes with gestures

This DIY Apple Pencil writes with gestures

Arduino Team — January 20, 2023

Launched in 2015, the Apple Pencil is a technology-packed stylus that allows users to write on iPad screens with varying pressure and angle, while communicating with very low latencies. Nekhil Ravi and Shebin Jose Jacob of Coders Café were inspired by this wearable technology to come up with their own pencil concept, except this one wouldn't need a screen to work.

The pair's writing utensil relies on recognizing certain gestures as letters, and once one has been detected, sends the result via USB or Bluetooth® to the host device. They started by collecting many samples of different letters and correlating them with motion change on the Arduino Nano 33 BLE Sense's built-in accelerometer. From there, they designed an impulse in Edge Impulse Studio to extract spectral features from time-series accelerometer data and feed it to a classification Keras neural network. The resulting model could accurately determine the correct letter of each gesture, making it suitable for deployment to the Nano 33 BLE Sense.

Before testing their new hardware inference code, a simple 3D printed enclosure was designed to fit around the board to resemble the real Apple Pencil. Additionally, the team created a simple website that could receive data from the board via BLE and display the corresponding letter in the browser window. To learn more about this project, you can watch their video below!