Penjam
Notes on the latest research in handwriting tracking
Handwriting Trajectory (2018)
Notes
- Online Character Recognition - tracking the pen movement
- Offline Character Recognition - picture of finished characters
- Goes into offline specifically which is a solved problem (see GPT-V below)
Testing GPT-V
Can take an image of my notebook with my handwriting and convert it into text perfectly.
Open Questions:
- How do we convert diagrams? Mermaid?
- Is this just good enough?
Improving Accuracy and Explainability of Online Handwriting Recognition (2022)
- Digipen data set was used
- “Offline handwriting recognition tries to recognize a handwritten text from its static digital image that has been produced at the time of writing. Online handwriting data is captured at run time, by some sort of sensor.”
- They open-sourced their modules
- Github Source
Couldn’t we do this with a camera?
Benefits:
- No research needed, no pen modifications, no special hardware - just works
- Can work for whiteboards
Drawbacks
- Power intensive (?)
- Invasive
Representation Learning for Tablet and Paper Domain Adaptation in Favor of Online Handwriting Recognition
- Meaty, trying to generalize a model on tablet and handwritten data.
There is a lack of online handwriting training data
- If we could make a cheap (cheap!) sensor that attaches to a pen simply
- Someone would probably use our training data..
dpoint
- https://opencv.org/ is interesting
Where I ended up
After all of this I decided a prototype with the Lamy nCode
It will allow me to validate if the core flow works well, and go from there. That is the big question every MVP should answer - is the core experience valuable?