Your shopping cart is empty!
![AI Virtual Keyboard with NVIDIA Jetson Nano AI Virtual Keyboard with NVIDIA Jetson Nano](https://static.cytron.io/image/cache/tutorial/ai-virtual-keyboard-with-nvidia-jetson-nano/AI-Virtual-Keyboard-with-NVIDIA-Jetson-Nano-800x320.png)
AI Virtual Keyboard with NVIDIA Jetson Nano
- Chun Boon Jian
- 20 Jan 2022
- 1323
Introduction
NVIDIA Jetson Nano is a “mini” computer that can perform Artificial Intelligence applications such as image classification, segmentation, speech processing, and object detection. In this tutorial, OpenCV and CVZone will be used to create an AI Virtual Keyboard by using Python language in Visual Studio Code with the NVIDIA Jetson Nano. Instead of using a physical keyboard, the AI Virtual Keyboard allows the user to type on the screen using a hand gesture with the help of a camera.
For this tutorial, I tried using NVIDIA Jetson Nano B01 and NVIDIA Jetson Nano 2GB (Discontinued), both gave the same result. Therefore, you can use either B01 or 2GB to create the AI Virtual Keyboard.
Hardware
- NVIDIA Jetson Nano Developer Kit (either one)
- USB Webcam
Module Preparation
These are the modules required to download on Jetson Nano before execute the Python code.
- OpenCV
- MediaPipe
- CVZone (must be 1.4.1 version)
- Pynput
- Open “Terminal”.
- Enter the following instructions line to install the required modules:
(For those haven’t installed pip)
$ sudo apt-get install python3-pip
(Start from here)
$ sudo pip3 install pip --upgrade
$ sudo pip3 install opencv_contrib_python
$ sudo apt install curl
$ sudo pip3 install opencv-python dataclasses
$ sudo apt install libcanberra-gtk-module libcanberra-gtk3-module
$ pip3 install cvzone == 1.4.1
$ pip install pynput
$ git clone https://github.com/PINTO0309/mediapipe-bin && cd mediapipe-bin
$ ./v0.8.5/numpy119x/mediapipe-0.8.5_cuda102-cp36-cp36m-linux_aarch64_numpy119x_jetsonnano_L4T32.5.1_download.sh
$ sudo pip3 install \
numpy-1.19.4-cp36-none-manylinux2014_aarch64.whl \
mediapipe-0.8.5_cuda102-cp36-none-linux_aarch64.whl
Python Coding
Copy the following Python code and paste in any Python IDE (Integrated Development Environment).
import cv2
from cvzone.HandTrackingModule import HandDetector
from time import sleep
import cvzone
from pynput.keyboard import Controller
cap = cv2.VideoCapture(0)
cap.set(3, 1280)
cap.set(4, 720)
detector = HandDetector(detectionCon=0.8)
keys = [["Q", "W", "E", "R", "T", "Y", "U", "I", "O", "P"],
["A", "S", "D", "F", "G", "H", "J", "K", "L", ";"],
["Z", "X", "C", "V", "B", "N", "M", ",", ".", "/"]]
finalText = ""
keyboard = Controller()
def drawALL(img, buttonList):
for button in buttonList:
x, y = button.pos
w, h = button.size
cvzone.cornerRect(
img, (button.pos[0], button.pos[1], button.size[0], button.size[1]), 20, rt=0, colorC=(32, 218, 165))
cv2.rectangle(img, button.pos, (x+w, y+h), (0, 0, 0), cv2.FILLED)
cv2.putText(img, button.text, (x+5, y+40),
cv2.FONT_HERSHEY_PLAIN, 3, (0, 215, 255), 5)
return img
class Button():
def __init__(self, pos, text, size=[50, 50]):
self.pos = pos
self.size = size
self.text = text
buttonList = []
for i in range(len(keys)):
for j, key in enumerate(keys[i]):
buttonList.append(Button([60 * j + 25, 60 * i + 30], key))
while True:
success, img = cap.read()
img = detector.findHands(img)
lmList, bboxInfo = detector.findPosition(img)
img = drawALL(img, buttonList)
if lmList:
for button in buttonList:
x, y = button.pos
w, h = button.size
# check is finger within the x&y axis button box
if x < lmList[8][0] < x + w and y < lmList[8][1] < y + h:
cv2.rectangle(img, button.pos, (x+w, y+h),
(204, 197, 194), cv2.FILLED)
cv2.putText(img, button.text, (x+5, y+40),
cv2.FONT_HERSHEY_PLAIN, 3, (88, 179, 197), 5)
l, _, _ = detector.findDistance(8, 12, img, draw=False)
print(l)
# when clicked
if l < 30:
keyboard.press(button.text)
cv2.rectangle(img, button.pos, (x+w, y+h),
(0, 215, 255), cv2.FILLED)
cv2.putText(img, button.text, (x+5, y+40),
cv2.FONT_HERSHEY_PLAIN, 3, (0, 0, 0), 5)
finalText += button.text
sleep(0.15)
cv2.rectangle(img, (50, 350), (600, 450), (88, 179, 197), cv2.FILLED)
cv2.putText(img, finalText, (60, 425),
cv2.FONT_HERSHEY_PLAIN, 5, (255, 255, 255), 5)
cv2.imshow("Image", img)
cv2.waitKey(1)
Instructions
- Move the tip of your index finger on the letter box to select the desired letter.
- Put the tip of your index finger and tip of the middle finger close together to press the key and type the letter out.
(**Note: Only put both of the fingertips close together when you want to type the letter out!) - You can type the letter out on any file, search bar, etc. using this program.
References
- AI Virtual Keyboard using OpenCV | Computer Vision | CVZone
- GitHub | mediapipe-bin
- OpenCV 4 + CUDA on Jetson Nano
Thanks for reading this tutorial. If you have any technical inquiries, please post at Cytron Technical Forum.