Intro to Software Reverse Engineering (Part 2)

Intro to Software Reverse Engineering (Part 2)

In our previous blog (part 1 of the “Intro to Software Reverse Engineering” series), we covered the basics of Reverse Engineering. In this blog (part 2 of the same series), we’ll be covering a walk-through of a challenge from crackmes.one. This challenge is an x86 binary executable which we’ll attempt to reverse engineer using OllyDBG

As before, the first step is to run the application.

software reverse engineering

When we run it, we are immediately presented with an error message box containing “Incorrect! Try again.”

software reverse engineering

This message box does not provide any useful information to us. Let’s proceed to load the executable on OllyDBG by navigating to File > Open.

software reverse engineering

Once it has finished loading, right-click on the window and search for all referenced strings.

software reverse engineering

A new window containing all the referenced strings pops up.

software reverse engineering

The “Incorrect! Try again.” string is familiar to us as we have seen it on the message box at the beginning. Double-click on the string and view the instructions on the CPU window.

software reverse engineering

On scrolling up a bit, we can observe an instruction at address 0040156F which calls a function and the following instructions compares the result and performs a conditional jump.

CALL C_File_C.00401501 // Call a function at address 00401501

TEST EAX,EAX // Performs bitwise AND operation and sets the Zero Flag (ZF) to 1 if EAX is zero

JNZ SHORT C_File_C.00401586 // If ZF is 0, jump to the address 004010586 which presents “Incorrect! Try again” message box. Otherwise, “Well done!…” message box is presented.

The Jump if Not Zero instruction effectively depends upon the value of EAX in the previous instruction. We’ll set a software breakpoint on the CALL instruction by pressing F2 on the line in OllyDBG.

software reverse engineering

Now we can resume execution of the program using the F9 key. The execution should pause at our breakpoint.

software reverse engineering

Press F7 to step into the CALL instruction.

software reverse engineering

Keep pressing F8 to step over instructions until we reach address 00401527 which contains the call to fopen with rb and password.bin as arguments. We’ll keep stepping over until we reach the address 00401544 which has the instruction to call fscanf which reads data from the file. Then we come across the fclose which closes the handle to the file. Then at address 401562, we see a strcmp instruction which compares two strings. Just before executing the strcmp instruction, we can see the two strings to be compared on the stack window, which is the one on the bottom right of the CPU window.

software reverse engineering

With this information, we are able to understand that the function obtains a valid handle to a file called password.bin, reads data from the file, closes the handle to the file and then compares the data read from the file with a string, which if it doesn’t match, sets EAX to FFFFFFFF (-1)., and if it matches, sets EAX to 0. For now, since the compared strings do not match, when we step over the strcmp instruction, EAX will be set to FFFFFFFF.

Software Reverse Engineering

This value of EAX is then used for the conditional jump instruction we saw before. We’ll now modify the password.bak file with notepad and add the same string used for comparison here.

software reverse engineering

Now, to verify, we’ll remove other breakpoints and set a new breakpoint at the strcmp instruction.

On observing the stack now, both strings in the stack are same.

software reverse engineering

On stepping over the “strcmp” instruction, observe that the value of EAX is now 0. This means the TEST EAX,EAX instruction now sets ZF to 0 and the conditional jump does not happen. 

software reverse engineering

On resuming execution, the “Well done!…” message box is presented to us.

software reverse engineering

We successfully managed to figure out how unknown executable binary works by disassembling it and observing the assembly instructions. In upcoming blogs (part 3), we’ll be going over some binaries which will be more difficult to reverse engineer compared to the ones we did in this blog.

By partnering with Redfox Security, you’ll get the best security and technical skills required to execute an effective and thorough penetration test. Our offensive security experts have years of experience assisting organizations in protecting their digital assets through penetration testing services. To schedule a call with one of our technical specialists, call 1-800-917-0850 now.

Redfox Security is a diverse network of expert security consultants with a global mindset and a collaborative culture. If you are looking to improve your organization’s security posture, contact us today to discuss your security testing needs. Our team of security professionals can help you identify vulnerabilities and weaknesses in your systems and provide recommendations to remediate them.

“Join us on our journey of growth and development by signing up for our comprehensive courses.