Control your Computer using Gestures

Motivation

I wanted to experiment further with Mediapipe's gesture detection and had just discovered pyautogui. I thought it would be cool to try to control my computer using just my webcam and gestures.

Challenge

Create a program that lets the user control their computer using only thier hands.

Solution

Using Mediapipe's gesture recongition a custom gesture model was created which allows the user to move their cursor, click, scroll, and more through just hand gestures.

Approach

In order to train my own gesture recongition model, I created a database of example gestures. I took approximately 100 pictures of several hand gestures to controll basic mouse functions. I then fed these images into a python script which trained a model to be used in another python script. This final script determines the location of the hand in the webcam camera and then projects its location to a pyautogui command which moves the cursor to that point on the screen. Pyautogui was also used to control basic commands such as click, scroll, and end the program.

Skills Demonstrated	Project Artifacts
OpenCV, Mediapipe, Gesture Recongition, Pyautogui	Github Repo

Benefits

This program can make using a computer more accessible, but was created as a learning experiment and likley has no real application.