Unlocking Gameplay Insights: How to Map User Inputs in Video Games using Computer Vision

Introduction
Gaming has become an integral part of modern entertainment, offering an immersive experience that transcends geographical boundaries. With the rise of mobile gaming, the demand for innovative and engaging gameplay has increased exponentially. In this tutorial, we will explore how to map user inputs for a mobile game using computer vision techniques, specifically focusing on the popular game Brawl Stars.

What We Are Building
Brawl Stars is a game with three main buttons: movement (blue), attack (red), and special attack (yellow). When the character is stagnant, all buttons are positioned directly above a similar, smaller button. However, when the character is moving, we can see that there is a button underneath the larger movement button. This is the same for both the attack and special attack buttons.

Step #1: Create a Roboflow Model
To begin, sign up for Roboflow and create an account. Next, go to workspaces and create a project. Customize the project name and annotation group to your choice. Make sure to make an object detection project. Draw the annotations using the bounding box feature in Roboflow. Repeat this step for each image. Make sure the key points are on each side of the weights.

Step #2: Create a Workflow
Using the model we created, we can use Roboflow Workflows, a low-code tool for building computer vision applications. To start, navigate to the Workflows tab on your Roboflow dashboard and create a workflow. Select the model you want to use. For this guide, we will use brawl-stars-buttons/3.

Code
Here is our code in full:

def on_prediction(res: dict, frame: VideoFrame) -> None:
    image = frame.image
    annotated_frame = image.copy()
    result = res['predictions']['predictions']
    if result is not None and hasattr(result, 'xyxy') and len(result.xyxy) > 0:
        attack = None
        joystick = None
        inner_buttons = []
        try:
            for i, class_name in enumerate(result.data['class_name']):
                if isinstance(class_name, np.ndarray):
                    class_name = class_name.item()  # Convert numpy string to Python string

                x_center = int((result.xyxy[i][0].item() + result.xyxy[i][2].item()) / 2)
                y_center = int((result.xyxy[i][1].item() + result.xyxy[i][3].item()) / 2)

                if class_name == 'attack':
                    attack = {'x': x_center, 'y': y_center}
                elif class_name == 'joystick':
                    joystick = {'x': x_center, 'y': y_center}
                elif class_name == 'inner_button':
                    inner_buttons.append({
                        'xyxy': result.xyxy[i],
                        'x': x_center,
                        'y': y_center
                    })
            if joystick and inner_buttons:
                # Find the middle point (use joystick if no attack)
                middle_x = joystick['x'] if not attack else (attack['x'] + joystick['x']) / 2
                middle_y = joystick['y'] if not attack else (attack['y'] + joystick['y']) / 2
                closest_middle = min(inner_buttons, key=lambda x: ((x['x'] - middle_x)**2 + (x['y'] - middle_y)**2)**0.5)
                joystick_direction, attack_direction = predict_direction(joystick, closest_middle, attack)
                # Display predicted directions
                cv2.putText(annotated_frame, f"Joystick: {joystick_direction}", (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
                if attack_direction:
                    cv2.putText(annotated_frame, f"Attack: {attack_direction}", (10, 70), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
                else:
                    cv2.putText(annotated_frame, "No Attack Detected", (10, 70), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
            # Annotate all detections
            annotated_frame = BOX_ANNOTATOR.annotate(scene=annotated_frame, detections=result)
            annotated_frame = LABEL_ANNOTATOR.annotate(scene=annotated_frame, detections=result)
        except Exception as e:
            print(f"Error processing frame: {e}")
    # Show the annotated frame
    cv2.imshow("frame", annotated_frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        return

**Conclusion**
In this guide, we were able to successfully deploy a Roboflow model and a workflow in order to map user inputs in games. We also learned how to use the detected information to map directions to multiple detections in our system. For more tutorials and content, feel free to check out some of our blogs.

**Frequently Asked Questions**

### Q1: What is Roboflow?
Roboflow is a platform that allows users to build and deploy computer vision applications using a low-code interface.

### Q2: What is a workflow in Roboflow?
A workflow in Roboflow is a series of steps that are executed in a specific order to perform a task, such as object detection or image classification.

### Q3: How do I create a Roboflow model?
To create a Roboflow model, you need to upload your dataset and annotate the images using the bounding box feature. Then, you can train the model using the Roboflow platform.

### Q4: What is the difference between a Roboflow model and a workflow?
A Roboflow model is a trained machine learning model that can perform a specific task, such as object detection. A workflow, on the other hand, is a series of steps that are executed in a specific order to perform a task.

### Q5: How do I deploy a Roboflow model?
To deploy a Roboflow model, you need to create a workflow that uses the trained model and then deploy the workflow to a specific environment, such as a cloud platform or a local machine.