SUPERCON 2022: Kuba Tyszko cracks encrypted software

[Kuba Tyszko], like many of us, has been a hacker since a young age. A first attempt at hacking with Grandpa's tractor could have been quickly quashed by his father, but it probably wasn't the last such incident. With a more recent interest in cracking encrypted apps, [Kuba] gives us an overview of some of the tools available to you for reading the encrypted secrets of apps that have something to hide. (Slides here, PDF.)

There can be all sorts of reasons for such apps to have an encrypted part, and that's not really the goal. One such application described by [Kuba] was a pre-trained machine learning model written in the R scripting language. If you are unfamiliar with R, it is commonly used for "science" type tasks data" and has a large fan base. Worth checking out. Anyway, the application binary took two command line arguments, one was the encrypted model blob and the second was the path to the test dataset for model checking .

The first thing [Kuba] suggests is to disable network access, just in case the app wants to "call home". We don't want that. The application was intended for Linux, so the first stop was to see what libraries it linked to using the ldd command. This indicated that it was related to OpenSSL, so it was a likely candidate for encryption support. Then running objdump gave some clues about the various components of the binary. It was determined to be doing something with 256-bit AES encryption. Now, after applying a bit of experience (or educated guesswork, if you prefer), the likely scenario is that the binary pulls the private key from somewhere into itself reads the encrypted blob file and passes it to libssl. Then the plain R script is passed to the R runtime, the model runs on the test data, and the results are gathered.

[Kuba]'s first attack method was to grab OpenSSL source code and insert strategic printf() function calls into the target functions. Then, using the LD_PRELOAD trick, the OpenSSL standard system library was replaced with the "fake" version with the printfs trojan. The result was the decryption function happily sending the plaintext R script directly to the terminal. No need to locate the private key!

Next, [Kuba] describes the "easy way" of freezing the binary, just like we could with an entire machine in years past, by reading it from a FIFO instead of a file , but never placing data on the other end. Then, with the read() call blocked, the binary is frozen and hopefully the private key is already in memory. Next, we use gcore to create a memory dump of the running application, which only requires knowledge of the process PID. Since the binary has already accessed the key and decrypted the secret model data, which it keeps in memory, the plaintext content will be in the main file and easily visible by simply opening it as a text file! After some searching, the R script code was visible. No special libraries are needed, just a handful of standard Linux commands.

With the shoe on the other foot, how can you protect your app against such a simple hacking process? Do you ride own crypto? It is a dangerous proposition and the consensus is not to do it. Preventing the FIFO attack could be as simple as using stat() to verify that the presented file is a real file. You can also statically link some critical libraries, to prevent the LD_PRELOAD attack, if possible. [Kuba] also suggests that the application can inspect all loaded shared objects using callbacks, to verify that the libraries are the ones expected.

The only way to be sure (and you can never be 100%) is to list all possible attack methods and mitigate each accordingly. There is no hack-proof method, you just have to make it as difficult as possible.