AI Virtual Keyboard with NVIDIA Jetson Nano

Chun Boon Jian
20 Jan 2022
861

Introduction

NVIDIA Jetson Nano is a “mini” computer that can perform Artificial Intelligence applications such as image classification, segmentation, speech processing, and object detection. In this tutorial, OpenCV and CVZone will be used to create an AI Virtual Keyboard by using Python language in Visual Studio Code with the NVIDIA Jetson Nano. Instead of using a physical keyboard, the AI Virtual Keyboard allows the user to type on the screen using a hand gesture with the help of a camera.

For this tutorial, I tried using NVIDIA Jetson Nano B01 and NVIDIA Jetson Nano 2GB (Discontinued), both gave the same result. Therefore, you can use either B01 or 2GB to create the AI Virtual Keyboard.

Hardware

NVIDIA Jetson Nano Developer Kit (either one)
- NVIDIA Jetson Nano Development Kit Rev.B01
- NVIDIA Jetson Nano 2GB Dev Kit-No Wireless Adapter (Discontinued)
USB Webcam

Module Preparation

These are the modules required to download on Jetson Nano before execute the Python code.

OpenCV
MediaPipe
CVZone (must be 1.4.1 version)
Pynput

Open “Terminal”.
Enter the following instructions line to install the required modules:

(For those haven’t installed pip)

$ sudo apt-get install python3-pip

(Start from here)

$ sudo pip3 install pip --upgrade

$ sudo pip3 install opencv_contrib_python

$ sudo apt install curl

$ sudo pip3 install opencv-python dataclasses

$ sudo apt install libcanberra-gtk-module libcanberra-gtk3-module

$ pip3 install cvzone == 1.4.1

$ pip install pynput

$ git clone https://github.com/PINTO0309/mediapipe-bin && cd mediapipe-bin

$ ./v0.8.5/numpy119x/mediapipe-0.8.5_cuda102-cp36-cp36m-linux_aarch64_numpy119x_jetsonnano_L4T32.5.1_download.sh

$ sudo pip3 install \

numpy-1.19.4-cp36-none-manylinux2014_aarch64.whl \

mediapipe-0.8.5_cuda102-cp36-none-linux_aarch64.whl

Python Coding

Copy the following Python code and paste in any Python IDE (Integrated Development Environment).

import cv2
from cvzone.HandTrackingModule import HandDetector
from time import sleep
import cvzone
from pynput.keyboard import Controller

cap = cv2.VideoCapture(0)
cap.set(3, 1280)
cap.set(4, 720)

detector = HandDetector(detectionCon=0.8)
keys = [["Q", "W", "E", "R", "T", "Y", "U", "I", "O", "P"],
        ["A", "S", "D", "F", "G", "H", "J", "K", "L", ";"],
        ["Z", "X", "C", "V", "B", "N", "M", ",", ".", "/"]]
finalText = ""

keyboard = Controller()


def drawALL(img, buttonList):

    for button in buttonList:
        x, y = button.pos
        w, h = button.size
        cvzone.cornerRect(
            img, (button.pos[0], button.pos[1], button.size[0], button.size[1]), 20, rt=0, colorC=(32, 218, 165))
        cv2.rectangle(img, button.pos, (x+w, y+h), (0, 0, 0), cv2.FILLED)
        cv2.putText(img, button.text, (x+5, y+40),
                    cv2.FONT_HERSHEY_PLAIN, 3, (0, 215, 255), 5)
    return img


class Button():
    def __init__(self, pos, text, size=[50, 50]):
        self.pos = pos
        self.size = size
        self.text = text


buttonList = []

for i in range(len(keys)):
    for j, key in enumerate(keys[i]):
        buttonList.append(Button([60 * j + 25, 60 * i + 30], key))

while True:
    success, img = cap.read()

    img = detector.findHands(img)
    lmList, bboxInfo = detector.findPosition(img)

    img = drawALL(img, buttonList)

    if lmList:
        for button in buttonList:
            x, y = button.pos
            w, h = button.size

            # check is finger within the x&y axis button box
            if x < lmList[8][0] < x + w and y < lmList[8][1] < y + h:
                cv2.rectangle(img, button.pos, (x+w, y+h),
                              (204, 197, 194), cv2.FILLED)
                cv2.putText(img, button.text, (x+5, y+40),
                            cv2.FONT_HERSHEY_PLAIN, 3, (88, 179, 197), 5)

                l, _, _ = detector.findDistance(8, 12, img, draw=False)
                print(l)

                # when clicked
                if l < 30:
                    keyboard.press(button.text)
                    cv2.rectangle(img, button.pos, (x+w, y+h),
                                  (0, 215, 255), cv2.FILLED)
                    cv2.putText(img, button.text, (x+5, y+40),
                                cv2.FONT_HERSHEY_PLAIN, 3, (0, 0, 0), 5)
                    finalText += button.text
                    sleep(0.15)

    cv2.rectangle(img, (50, 350), (600, 450), (88, 179, 197), cv2.FILLED)
    cv2.putText(img, finalText, (60, 425),
                cv2.FONT_HERSHEY_PLAIN, 5, (255, 255, 255), 5)

    cv2.imshow("Image", img)
    cv2.waitKey(1)

Instructions

Move the tip of your index finger on the letter box to select the desired letter.
Put the tip of your index finger and tip of the middle finger close together to press the key and type the letter out.
(**Note: Only put both of the fingertips close together when you want to type the letter out!)
You can type the letter out on any file, search bar, etc. using this program.

References

Thanks for reading this tutorial. If you have any technical inquiries, please post at Cytron Technical Forum.

Tags