Gracefully handle dex file decompilation errors, skip problematic dex

[AbhiTheModder]

Some protectors/packers intentionally add useless/unwanted dex files under apk to hinder tools like apktool/apkeditor to fail during decompilation as they're not correct pure dex files(classes0.dex, there's no such thing as classes0 in an apk file, generally dex files start with classes.dex, classes2.dex and so on..) which leads to the tools being not able to complete apk decompilation.

I: [DECOMPILE] Baksmali: classes0.dex

ERROR:
org.jf.util.ExceptionWithContext: Encountered small uint that is out of range at offset 0x70
        at org.jf.dexlib2.dexbacked.DexBuffer.readSmallUint(DexBuffer.java:59)
        at org.jf.dexlib2.dexbacked.model.DexStringSection.load(DexStringSection.java:44)
        at org.jf.dexlib2.dexbacked.DexBackedDexFile.<init>(DexBackedDexFile.java:98)
        at org.jf.dexlib2.dexbacked.DexBackedDexFile.<init>(DexBackedDexFile.java:204)
        at com.reandroid.apkeditor.smali.SmaliDecompiler.getInputDexFile(SmaliDecompiler.java:179)
        at com.reandroid.apkeditor.smali.SmaliDecompiler.disassembleWithJesusFrekeLib(SmaliDecompiler.java:119)
        at com.reandroid.apkeditor.smali.SmaliDecompiler.decodeDex(SmaliDecompiler.java:64)
        at com.reandroid.apk.DexDecoder.decodeDex(DexDecoder.java:29)
        at com.reandroid.apkeditor.smali.SmaliDecompiler.decodeDex(SmaliDecompiler.java:71)
        at com.reandroid.apk.ApkModuleDecoder.decodeDexFiles(ApkModuleDecoder.java:113)
        at com.reandroid.apk.ApkModuleDecoder.decode(ApkModuleDecoder.java:58)
        at com.reandroid.apkeditor.decompile.Decompiler.runCommand(Decompiler.java:62)
        at com.reandroid.apkeditor.Options.runCommand(Options.java:59)
        at com.reandroid.apkeditor.Main.run(Main.java:136)
        at com.reandroid.apkeditor.Main.execute(Main.java:72)
        at com.reandroid.apkeditor.Main.main(Main.java:57)

This commit enhances the SmaliDecompiler to gracefully handle errors encountered during the decompilation of individual dex files. Previously, encountering a malformed or invalid dex file would halt the entire decompilation process.

Upon encountering an error, instead of halting, it now logs the error and skips the problematic dex file, continuing the decompilation process with the other files.

After:

15.223 I: [DECOMPILE] Baksmali: classes.dex                                                                                                             
35.430 I: [DECOMPILE] Baksmali: classes0.dex
35.431 I: [DECOMPILE] Error decompiling dex file: classes0.dex ,Encountered small uint that is out of range at offset 0x70
35.432 I: [DECOMPILE] Baksmali: classes2.dex
51.508 I: [DECOMPILE] Baksmali: classes3.dex
01:03.749 I: [DECOMPILE] Baksmali: classes4.dex
01:12.818 I: [DECOMPILE] Baksmali: classes5.dex
01:12.932 I: [DECOMPILE] Baksmali: classes6.dex
01:14.688 I: [DECOMPILE] Baksmali: classes7.dex
01:14.716 I: [DECOMPILE] Baksmali: classes8.dex
01:14.834 I: [DECOMPILE] Extracting root files ...

[AbhiTheModder]

We could think about adding a flag for this in the decompile options, it might give more flexibility. For now, I stuck with the simpler solution since it works without adding complexity.

P.S.: Another reason being naming stuff is hard! :haha: I couldn't settle on a good name for the flag.

[AbhiTheModder]

While I was at it, I discovered another similar PR in google/smali addressing other types of invalid dex types, so I've included that as well.

P.S: Oops typo in commit message (or of) :)

Edit: Okay that was wrong assumption, built baksmali from source and verified on the same file on which APKEditor throws ExceptionWithContext error, baksmali throws ArrayIndexOutOfBoundsException so they're equivalent, no need to add more un-necessary code :)

P.S: Consider merging PR with squash & merge to avoid wrong commit going to HEAD :)

[REAndroid]

Thank you,
I encountered similar issue, the problem is only with classes0.dex and i believe it should be fixed on ARSCLib side InputSource#getDexNumber.

[REAndroid]

Check this commit

[AbhiTheModder]

Thank you, I encountered similar issue, the problem is only with classes0.dex and i believe it should be fixed on ARSCLib side InputSource#getDexNumber.

Awesome, that will suffice as well.