Using Win32 API with Rust: A Step-by-Step Guide

Using Win32 API with Rust

The Win32 API is a cornerstone of Windows programming, offering a wide range of functions to interact directly with the Windows operating system. Traditionally accessed through C or C++, the Win32 API can also be harnessed using Rust, a language renowned for its safety, concurrency, and performance. Combining Rust with the Win32 API empowers developers to build robust Windows applications with reduced risk of common bugs such as memory safety issues.

In this post, we’ll walk through how to set up a Rust project that calls the Win32 API, specifically demonstrating how to use the MessageBoxW function to display a Windows message box.

Getting Started with Rust and Win32 API

We will use the winapi crate, which provides Rust bindings for the Win32 API.

Step 1: Create a New Rust Project

				
					cargo new –bin api-test
				
			
  • Cargo is a package manager for Rust that you can use to easily manage Rust projects.
  • New – is used to create a new project
  • –bin – is a flag to let the compiler know that this code will result in an executable windows binary
  • Api-test – is the name of the project

This will create a folder named ‘api-test’ in the directory which will contain main.rs file in the ‘src’ folder and a cargo.toml file which will contain all the dependencies for the project.

Now, let’s add Winapi crate dependency in the project. There’s two ways we can do this first we run a command after navigating to the ‘api-test’ folder.

Step 2: Add the Winapi Dependency

Navigate into the project directory and add the winapi crate with the winuser feature, which contains user interface-related APIs:

				
					cd api-test 
cargo add winapi –features winuser
				
			

Alternatively, add this manually to your Cargo.toml file:

				
					[dependencies]
winapi = { version = “0.3.9”, features = [“winuser”] }
				
			

Understanding the Win32 API MessageBoxW

The MessageBoxW function creates a message box window with customizable text, title, buttons, and icons. Its signature in C looks like this:

To read more about the MessageBoxW api, click here  

				
					int MessageBoxW(
  [in, optional] HWND    hWnd,
  [in, optional] LPCWSTR lpText,
  [in, optional] LPCWSTR lpCaption,
  [in]           UINT    uType
);
				
			
  • HWND – This is a handle to the window’s owner. If it’s NULL, the message box has no owner window.
  • LPCTSTR – Message to be displayed in the window
  • LPCTSTR – Caption of the window to be created.
  • UINT – The window’s contents and behaviour can define the number of buttons and icons it holds.

Let’s start with the code:

				
					extern crate winapi;
use winapi::um::winuser::{MessageBoxW, MB_ABORTRETRYIGNORE, MB_ICONQUESTION, IDABORT, IDRETRY, IDIGNORE};
use std::ptr::null_mut;
				
			
  • 1st line includes the Winapi crate in the current program to be used
  • 2nd line imports the functions and definitions into the scope of the program
  • 3rd line imports the null_mut constant in the scope to represent a null pointer
				
					fn main(){}
				
			
  • The entry point for a Rust program
				
					loop {}
				
			
  • Creates an infinite loop in Rust (it’ll make more sense on why we use an infinite loop in a bit)
				
					let title: vec<u16> = “messageboxw\0”.encode_utf16().collect();
let content: vec<u16> = "this is a messageboxw test\0".encode_utf16().collect();

				
			
  • Defines variables named title and content containing messages for the Message Box
  • .encode_utf16() – this is a function that encodes the string in utf16 format and returns an iterator (you can ignore what iterator means for this tutorial)
  • .collect() – collects the iterator in the respective variables.
				
					unsafe{}
				
			
  • In Rust, we use the keyword when calling unsafe functions. This basically means these functions don’t guarantee memory safety as the built-in rust functions do.
				
					let reply = messageboxw(null_mut(), content.as_ptr(), title.as_ptr(), MB_ABORTRETRYIGNORE | MB_ICONQUESTION);
				
			
  • Defines a variable named reply which contains the value returned by the MessageBoxW function.
  • MessageBoxW – is a function provided by WinAPI crate, which enables us to create a message box window
  • Null_mut() – is a reference to a null pointer is rust. This basically means NULL
  • as_ptr() – is a raw pointer to the contents of the variable content
  • as_ptr() – is a raw pointer to the contents of the variable title
  • MB_ABORTRETRYIGNORT – this value means our message box will contain three buttons.
win32 api with rust
  • MB_ICONQUESTION – this will include a question icon in the message box
win32 api with rust
				
					match reply {
                IDRETRY => {
                    println!("You pressed Retry");
                    continue;
                },
                IDIGNORE => {
                    println!("You pressed Ignore");
                    continue;
                },
                IDABORT => {
                    println!("Okay.... Aborting now!");
                    break;
                },
                _ => {
                    unreachable!();
                }

				
			
  • Match – Match is a control flow construct for pattern matching used to perform actions based on the reply from the MessageBoxW function.
  • IDRETRY – Pressing the RETRY button returns IDRETRY, causing the loop to continue indefinitely.
  • IDIGNORE – Pressing the IGNORE button returns IDIGNORE, causing the loop to continue indefinitely.
  • IDABORT – Pressing the ABORT button returns IDABORT, causing the loop to exit.
  • _ => – is a catch-all statement, which matches everything which doesn’t match IDRETRY, IDIGNORE and IDABORT. This will never happen in our code as our MessageBoxW is only capable of providing these values in this context. That’s why we use the unreachable!() macro; it tells the compiler that this match statement is impossible to reach or is logically unreachable.

Full Code

				
					extern crate winapi;
use winapi::um::winuser::{MessageBoxW, MB_ABORTRETRYIGNORE, MB_ICONQUESTION, IDABORT, IDRETRY, IDIGNORE};
use std::ptr::null_mut;

fn main(){
    loop {
        let title: Vec<u16> = "MessageBoxW\0".encode_utf16().collect();
        let content: Vec<u16> = "This is a MessageBoxW test\0".encode_utf16().collect();
        unsafe{
            let reply = MessageBoxW(null_mut(), content.as_ptr(), title.as_ptr(), MB_ABORTRETRYIGNORE | MB_ICONQUESTION);
            match reply {
                IDRETRY => {
                    println!("You pressed Retry");
                    continue;
                },
                IDIGNORE => {
                    println!("You pressed Ignore");
                    continue;
                },
                IDABORT => {
                    println!("Okay.... Aborting now!");
                    break;
                },
                _ => {
                    unreachable!();
                }
            }
        }
    }
}

				
			

Running the code

TL;DR

Combining Rust with the Win32 API lets you build efficient, safe, and native Windows applications. Using the winapi crate, you can easily call Windows functions like MessageBoxW. This tutorial showed how to set up a Rust project, add dependencies, and implement a message box with button handling.

Redfox Security is a diverse network of expert security consultants with a global mindset and a collaborative culture. If you are looking to improve your organization’s security posture, contact us today to discuss your security testing needs. Our team of security professionals can help you identify vulnerabilities and weaknesses in your systems and provide recommendations to remediate them.

Join us on our journey of growth and development by signing up for our comprehensive courses.