dexOpt & odex

作者: 小马要加油 | 来源:发表于2023-09-19 17:09 被阅读0次

（基于android12分析和测试）

一、现存问题

Android早期是aot的方式先编译成机器码，然后再运行的，这样会导致安装时间变长，后面的版本改成jit编译方式，在运行时编译，这样会导致运行速度较慢。android7.0之后的版本支持jit+oat的方式编译，支持app配置baseline-profile的方式设置热点代码。

二、理论基础

1、ART 执行方式

最初安装应用时不进行任何 AOT 编译。应用前几次运行时，系统会对其进行解译，并对经常执行的方法进行 JIT 编译。
当设备闲置和充电时，编译守护程序会运行，以便根据在应用前几次运行期间生成的配置文件对常用代码进行 AOT 编译。
下一次重新启动应用时将会使用配置文件引导型代码，并避免在运行时对已经过编译的方法进行 JIT 编译。在应用后续运行期间经过 JIT 编译的方法将会添加到配置文件中，然后编译守护程序将会对这些方法进行 AOT 编译。

1695197336986.png

2、编译选项

dex2oat工具可以将dex文件生成vdex，odex或者.art文件。其中：

.dex
.java->.class->.dex （apk解压后的问题）
java被编译成.class之后，使用d8工具（以前是dx）将class文件合成dex文件，dex一般是jar的50%大小。然后被打包成单个 .apk 文件。.dex 文件可以通过自动转换用 Java 编程语言编写的编译应用程序来创建。
.odex一种文件格式
.java->.class->.dex->.oat
过 AOT 编译的方法代码,ART可以直接用的机器码。

.vdex
dex->.vdex
对dex文件进行初步优化，调用dexOpt方法，转成vdex文件（文件名后缀依然是.dex），只是小小的优化了操作码，
其中odex的文件是可以直接被运行的。生成那种类型的文件依赖dex2Oat工具，dex2Oat依赖一个核心参数“编译过滤器”

编译过滤器：(android 官方sdk android8.0之后没有再更新)

verify：仅运行 DEX 代码验证。
quicken：（从 Android 12 开始已移除）运行 DEX 代码验证，并优化一些 DEX 指令，以获得更好的解译器性能。（我在12还是看到有这样配置）
speed：运行 DEX 代码验证，并对所有方法进行 AOT 编译。
speed-profile：运行 DEX 代码验证，并对配置文件中列出的方法进行 AOT 编译。

实际上最新的android 12代码中增加了一些。

//android_12/art/libartbase/base/compiler_filter.h
enum Filter {
  kAssumeVerified,      // Skip verification but mark all classes as verified anyway.
  kExtract,             // Delay verication to runtime, do not compile anything.
  kVerify,              // Only verify classes.
  kSpaceProfile,        // Maximize space savings based on profile.
  kSpace,               // Maximize space savings.
  kSpeedProfile,        // Maximize runtime performance based on profile.
  kSpeed,               // Maximize runtime performance.
  kEverythingProfile,   // Compile everything capable of being compiled based on profile.
  kEverything,          // Compile everything capable of being compiled.
};
 
 
 
//android_12/art/libartbase/base/compiler_filter.cc
std::string CompilerFilter::NameOfFilter(Filter filter) {
  switch (filter) {
    case CompilerFilter::kAssumeVerified: return "assume-verified";
    case CompilerFilter::kExtract: return "extract";
    case CompilerFilter::kVerify: return "verify";
    case CompilerFilter::kSpaceProfile: return "space-profile";
    case CompilerFilter::kSpace: return "space";
    case CompilerFilter::kSpeedProfile: return "speed-profile";
    case CompilerFilter::kSpeed: return "speed";
    case CompilerFilter::kEverythingProfile: return "everything-profile";
    case CompilerFilter::kEverything: return "everything";
  }
  UNREACHABLE();
}

从这里可以得知，quicken被移除了，如果配置了重定向到verify 。配置speed可以将所有方法进行oat，从而加速代码的运行速度。配置speed-profile，可以选择性的让配置的方法进行AOT编译。

3、强制编译命令

系统支持用命令执行odex编译

命令：

adb shell cmd package compile

基于配置文件：

adb shell cmd package compile -m speed-profile -f my-package

全面编译：

adb shell cmd package compile -m speed -f my-package

代码执行流程：

1695197534122.png

命令执行后会生成odex和vdex文件，放置在data/app/package/oat/arm[64]/xxx

重启进程后可以使用命令查：

/proc/pid/maps/ |grep "odex"

这样就看到应用加载了odex加载到了内存。

4、手动执行dex2Oat

android系统是自带dex2oat工具的，直接在平台执行dex2oat 命令可以直接生成对应的文件，默认位speed编译

dex2oat --dex-file=a.dex --oat-file=./oat/arm64/base.odex

不过我企图生成odex去覆盖原来的odex失败了。简单看了ART执行的逻辑，应该是校验不通过导致。（没有仔细研究，只是简单看看代码+推测）。

后面发现如果用PackageManagerService（pkms）去生成一个odex就可以用。于是我加了点log查看一样的命令参数，使用pkms同款编译参数后就可以用了。放上研究了一天的命令参数

dex2oat32 --dex-file=./data/app/xxx/xxx.dex --oat-file=/data/app/xxx/oat/arm/xxx.odex --classpath-dir=/data/app/xxx --class-loader-context=PCL[]{PCL[/system/framework/org.apache.http.legacy.jar]} --instruction-set=arm --instruction-set-features=default --instruction-set-variant=cortex-a7 --compiler-filter=speed --compilation-reason=cmdline --max-image-block-size=524288 --resolve-startup-const-strings=true --generate-mini-debug-info --runtime-arg -Xdeny-art-apex-data-files  --runtime-arg -Xtarget-sdk-version:31 --runtime-arg -Xhidden-api-policy:enabled -j4 --runtime-arg -Xms64m --runtime-arg -Xmx512m --compile-individually

三、PKMS 代码执行逻辑

1、pkms执行逻辑

1695200019816.png

int performDexOpt(AndroidPackage pkg, @NonNull PackageSetting pkgSetting,
        String[] instructionSets, CompilerStats.PackageStats packageStats,
        PackageDexUsage.PackageUseInfo packageUseInfo, DexoptOptions options) {
 
    if (PLATFORM_PACKAGE_NAME.equals(pkg.getPackageName())) {
        throw new IllegalArgumentException("System server dexopting should be done via "
                + " DexManager and PackageDexOptimizer#dexoptSystemServerPath");
    }
    if (pkg.getUid() == -1) {
        throw new IllegalArgumentException("Dexopt for " + pkg.getPackageName()
                + " has invalid uid.");
    }
    if (!canOptimizePackage(pkg)) {//过滤不允许oat的
        return DEX_OPT_SKIPPED;
    }
    synchronized (mInstallLock) {
        final long acquireTime = acquireWakeLockLI(pkg.getUid());
        try {
            return performDexOptLI(pkg, pkgSetting, instructionSets,
                    packageStats, packageUseInfo, options);
        } finally {
            releaseWakeLockLI(acquireTime);
        }
    }
}
 
// 进入performDexOptLI(pkg, pkgSetting, instructionSets, packageStats, packageUseInfo, options)
 
private int performDexOptLI(AndroidPackage pkg, @NonNull PackageSetting pkgSetting,
        String[] targetInstructionSets, CompilerStats.PackageStats packageStats,
        PackageDexUsage.PackageUseInfo packageUseInfo, DexoptOptions options) {
...
 
    int result = DEX_OPT_SKIPPED;
    for (int i = 0; i < paths.size(); i++) {
        // Skip paths that have no code.
        if (!pathsWithCode[i]) {
            continue;
        }
        if (classLoaderContexts[i] == null) {
            throw new IllegalStateException("Inconsistent information in the "
                    + "package structure. A split is marked to contain code "
                    + "but has no dependency listed. Index=" + i + " path=" + paths.get(i));
        }
 
        // Append shared libraries with split dependencies for this split.
        String path = paths.get(i);
        if (options.getSplitName() != null) {
            // We are asked to compile only a specific split. Check that the current path is
            // what we are looking for.
            if (!options.getSplitName().equals(new File(path).getName())) {
                continue;
            }
        }
 
        String profileName = ArtManager.getProfileName(
                i == 0 ? null : pkg.getSplitNames()[i - 1]); //找profile文件
 
        ...
 
        final String compilerFilter = getRealCompilerFilter(pkg,
            options.getCompilerFilter(), isUsedByOtherApps);//对一些过滤编译器做调整
        ...
 
        for (String dexCodeIsa : dexCodeInstructionSets) {
            int newResult = dexOptPath(pkg, pkgSetting, path, dexCodeIsa, compilerFilter,
                    profileAnalysisResult, classLoaderContexts[i], dexoptFlags, sharedGid,
                    packageStats, options.isDowngrade(), profileName, dexMetadataPath,
                    options.getCompilationReason());
        ...
    return result;
}
 
//进入dexOptPath
 
private int dexOptPath(AndroidPackage pkg, @NonNull PackageSetting pkgSetting, String path,
        String isa, String compilerFilter, int profileAnalysisResult, String classLoaderContext,
        int dexoptFlags, int uid, CompilerStats.PackageStats packageStats, boolean downgrade,
        String profileName, String dexMetadataPath, int compilationReason) {
   ...
    String oatDir = getPackageOatDirIfSupported(pkg,
            pkgSetting.getPkgState().isUpdatedSystemApp());
 
    ...
        mInstaller.dexopt(path, uid, pkg.getPackageName(), isa, dexoptNeeded, oatDir,
                dexoptFlags, compilerFilter, pkg.getVolumeUuid(), classLoaderContext,
                seInfo, false /* downgrade*/, pkg.getTargetSdkVersion(),
                profileName, dexMetadataPath,
                getAugmentedReasonName(compilationReason, dexMetadataPath != null)); //调用Installer 执行dexOpt过程
 
        ...
        return DEX_OPT_PERFORMED;
    } catch (InstallerException e) {
        Slog.w(TAG, "Failed to dexopt", e);
        return DEX_OPT_FAILED;
    }
}

调用performDexOpt需要传入以下参数

performDexOpt(AndroidPackage pkg, @NonNull PackageSetting pkgSetting,

String[] instructionSets,

CompilerStats.PackageStats packageStats,

PackageDexUsage.PackageUseInfo packageUseInfo,

DexoptOptions options)

其中DexoptOptions 定义了编译的可选项，其构造方法如下，有个比较重要的属性compilationReason，构造方法2也是通过reason获取compilerFilter的。

//构造1
public DexoptOptions(String packageName, String compilerFilter, int flags) {
    this(packageName, /*compilationReason*/ -1, compilerFilter, /*splitName*/ null, flags);
}
 
//构造2
public DexoptOptions(String packageName, int compilationReason, int flags) {
    this(packageName, compilationReason, getCompilerFilterForReason(compilationReason),
            /*splitName*/ null, flags);
}
 
//构造3
public DexoptOptions(String packageName, int compilationReason, String compilerFilter,
            String splitName, int flags) {
    int validityMask =
            DEXOPT_CHECK_FOR_PROFILES_UPDATES |
            DEXOPT_FORCE |
            DEXOPT_BOOT_COMPLETE |
            DEXOPT_ONLY_SECONDARY_DEX |
            DEXOPT_ONLY_SHARED_DEX |
            DEXOPT_DOWNGRADE |
            DEXOPT_AS_SHARED_LIBRARY |
            DEXOPT_IDLE_BACKGROUND_JOB |
            DEXOPT_INSTALL_WITH_DEX_METADATA_FILE |
            DEXOPT_FOR_RESTORE;
    if ((flags & (~validityMask)) != 0) {
        throw new IllegalArgumentException("Invalid flags : " + Integer.toHexString(flags));
    }
 
    mPackageName = packageName;
    mCompilerFilter = compilerFilter;
    mFlags = flags;
    mSplitName = splitName;
    mCompilationReason = compilationReason;
}

通过reason获取compilerFilter的。也就是从这个通过prop定义的值完成设置编译过滤器。

public static String getCompilerFilterForReason(int reason) {
   return getAndCheckValidity(reason);
}

/ Load the property for the given reason and check for validity. This will throw an
// exception in case the reason or value are invalid.
private static String getAndCheckValidity(int reason) {
   String sysPropValue = SystemProperties.get(getSystemPropertyName(reason));
   if (sysPropValue == null || sysPropValue.isEmpty()
           || !(sysPropValue.equals(DexoptOptions.COMPILER_FILTER_NOOP)
                   || DexFile.isValidCompilerFilter(sysPropValue))) {
       throw new IllegalStateException("Value \"" + sysPropValue +"\" not valid "
               + "(reason " + REASON_STRINGS[reason] + ")");
   } else if (!isFilterAllowedForReason(reason, sysPropValue)) {
       throw new IllegalStateException("Value \"" + sysPropValue +"\" not allowed "
               + "(reason " + REASON_STRINGS[reason] + ")");
   }

   return sysPropValue;
}

PKMS 中定义了14中编译原因，对应了每种触发dexOpt的原因，和执行dexOpt定义的编译过滤器。

// Compilation reasons.
public static final int REASON_FIRST_BOOT = 0;
public static final int REASON_BOOT_AFTER_OTA = 1;
public static final int REASON_POST_BOOT = 2;
public static final int REASON_INSTALL = 3;
public static final int REASON_INSTALL_FAST = 4;
public static final int REASON_INSTALL_BULK = 5;
public static final int REASON_INSTALL_BULK_SECONDARY = 6;
public static final int REASON_INSTALL_BULK_DOWNGRADED = 7;
public static final int REASON_INSTALL_BULK_SECONDARY_DOWNGRADED = 8;
public static final int REASON_BACKGROUND_DEXOPT = 9;
public static final int REASON_AB_OTA = 10;
public static final int REASON_INACTIVE_PACKAGE_DOWNGRADE = 11;
public static final int REASON_CMDLINE = 12;
public static final int REASON_SHARED = 13;

2、BackgroundDexOptService

BackgroundDexOptService 是一个JobService，用于定期执行任务。在SystemServer#startOtherService()方法中启动，启动后执行两个任务。

任务一：监听开机广播，在开机10分钟-60分钟内完成post-boot的dex优化

任务二：每隔一天执行一次idle 场景下dex优化。

1695200395519.png

//SystemServer.java 
private void startOtherServices(@NonNull TimingsTraceAndSlog t) {
...
t.traceBegin("StartBackgroundDexOptService");
try {
    BackgroundDexOptService.schedule(context);
} catch (Throwable e) {
    reportWtf("starting StartBackgroundDexOptService", e);
}
t.traceEnd();
...
}

// BackgroundDexOptService.java
public static void schedule(Context context) {
    if (isBackgroundDexoptDisabled()) {//读属性"pm.dexopt.disable_bg_dexopt" ,目前是false
        return;
    }
 
    final JobScheduler js = context.getSystemService(JobScheduler.class);
 
    // Schedule a one-off job which scans installed packages and updates
    // out-of-date oat files. Schedule it 10 minutes after the boot complete event,
    // so that we don't overload the boot with additional dex2oat compilations.
    context.registerReceiver(new BroadcastReceiver() {
        @Override
        public void onReceive(Context context, Intent intent) {
            js.schedule(new JobInfo.Builder(JOB_POST_BOOT_UPDATE, sDexoptServiceName)//BackgroundDexOptService 的第一个任务
                    .setMinimumLatency(TimeUnit.MINUTES.toMillis(10)) //最短执行时间10min
                    .setOverrideDeadline(TimeUnit.MINUTES.toMillis(60)) //最迟执行时间：60mins
                    .build());
            context.unregisterReceiver(this);
            if (DEBUG) {
                Slog.i(TAG, "BootBgDexopt scheduled");
            }
        }
    }, new IntentFilter(Intent.ACTION_BOOT_COMPLETED));//监听开机启动广播
 
    // Schedule a daily job which scans installed packages and compiles
    // those with fresh profiling data.
    js.schedule(new JobInfo.Builder(JOB_IDLE_OPTIMIZE, sDexoptServiceName) //BackgroundDexOptService 的第二个任务
                .setRequiresDeviceIdle(true) //设备在idle状态
                .setRequiresCharging(true) //充电中
                .setPeriodic(IDLE_OPTIMIZATION_PERIOD) //执行周期 一天
                .build());
 
    if (DEBUG) {
        Slog.d(TAG, "BgDexopt scheduled");
    }
}
 
 
//关注第一个job：到点执行进入onStartJob方法
public boolean onStartJob(JobParameters params) {
    if (DEBUG) {
        Slog.i(TAG, "onStartJob");
    }
 
    // NOTE: PackageManagerService.isStorageLow uses a different set of criteria from
    // the checks above. This check is not "live" - the value is determined by a background
    // restart with a period of ~1 minute.
    PackageManagerService pm = (PackageManagerService)ServiceManager.getService("package");
    if (pm.isStorageLow()) {//Environment.getDataDirectory().getUsableSpace() < getMemoryLowThreshold(); 为data空间的20%或者低于500MB
        Slog.i(TAG, "Low storage, skipping this run");
        return false;
    }
 
    final ArraySet<String> pkgs = pm.getOptimizablePackages();
    if (pkgs.isEmpty()) {
        Slog.i(TAG, "No packages to optimize");
        return false;
    }
 
    mThermalStatusCutoff =
        SystemProperties.getInt("dalvik.vm.dexopt.thermal-cutoff", THERMAL_CUTOFF_DEFAULT);//2
 
    boolean result;
    if (params.getJobId() == JOB_POST_BOOT_UPDATE) {
        result = runPostBootUpdate(params, pm, pkgs);//进入这里
    } else {
        result = runIdleOptimization(params, pm, pkgs);
    }
 
    return result;
}
 
//执行post-boot更新
private boolean runPostBootUpdate(final JobParameters jobParams,
        final PackageManagerService pm, final ArraySet<String> pkgs) {
    if (mExitPostBootUpdate.get()) {
        // This job has already been superseded. Do not start it.
        return false;
    }
    new Thread("BackgroundDexOptService_PostBootUpdate") {//新起一个线程执行postBootUpdate
        @Override
        public void run() {
            postBootUpdate(jobParams, pm, pkgs);
        }
 
    }.start();
    return true;
}
 
//子线程运行执行所有可优化包的dex优化
private void postBootUpdate(JobParameters jobParams, PackageManagerService pm,
        ArraySet<String> pkgs) {
    final BatteryManagerInternal batteryManagerInternal =
            LocalServices.getService(BatteryManagerInternal.class);
    final long lowThreshold = getLowStorageThreshold(this);
 
    mAbortPostBootUpdate.set(false);
 
    ArraySet<String> updatedPackages = new ArraySet<>();
    for (String pkg : pkgs) {
        if (mAbortPostBootUpdate.get()) {
            // JobScheduler requested an early abort.
            return;
        }
        if (mExitPostBootUpdate.get()) {
            // Different job, which supersedes this one, is running.
            break;
        }
        if (batteryManagerInternal.getBatteryLevelLow()) {//低电量不执行
            // Rather bail than completely drain the battery.
            break;
        }
        long usableSpace = mDataDir.getUsableSpace();
        if (usableSpace < lowThreshold) {//存储不足
            // Rather bail than completely fill up the disk.
            Slog.w(TAG, "Aborting background dex opt job due to low storage: " +
                    usableSpace);
            break;
        }
        if (DEBUG) {
            Slog.i(TAG, "Updating package " + pkg);
        }
 
        // Update package if needed. Note that there can be no race between concurrent
        // jobs because PackageDexOptimizer.performDexOpt is synchronized.
 
        // checkProfiles is false to avoid merging profiles during boot which
        // might interfere with background compilation (b/28612421).
        // Unfortunately this will also means that "pm.dexopt.boot=speed-profile" will
        // behave differently than "pm.dexopt.bg-dexopt=speed-profile" but that's a
        // trade-off worth doing to save boot time work.
        int result = pm.performDexOptWithStatus(new DexoptOptions(//进入PKMS
                pkg,
                PackageManagerService.REASON_POST_BOOT,//原因
                DexoptOptions.DEXOPT_BOOT_COMPLETE));
        if (result == PackageDexOptimizer.DEX_OPT_PERFORMED)  {
            updatedPackages.add(pkg);
        }
    }
    notifyPinService(updatedPackages);
    notifyPackagesUpdated(updatedPackages);
    // Ran to completion, so we abandon our timeslice and do not reschedule.
    jobFinished(jobParams, /* reschedule */ false);
}
 
进入PKMS 的performDexOptWithStatus方法。
/* package */ int performDexOptWithStatus(DexoptOptions options) {
    return performDexOptTraced(options);
}
 
 
private int performDexOptTraced(DexoptOptions options) {
    Trace.traceBegin(TRACE_TAG_PACKAGE_MANAGER, "dexopt");
    try {
        return performDexOptInternal(options);
    } finally {
        Trace.traceEnd(TRACE_TAG_PACKAGE_MANAGER);
    }
}
 
 
private int performDexOptInternal(DexoptOptions options) {
    AndroidPackage p;
    PackageSetting pkgSetting;
    synchronized (mLock) {
        p = mPackages.get(options.getPackageName());
        pkgSetting = mSettings.getPackageLPr(options.getPackageName());
        if (p == null || pkgSetting == null) {
            // Package could not be found. Report failure.
            return PackageDexOptimizer.DEX_OPT_FAILED;
        }
        mPackageUsage.maybeWriteAsync(mSettings.getPackagesLocked());
        mCompilerStats.maybeWriteAsync();
    }
    final long callingId = Binder.clearCallingIdentity();
    try {
        synchronized (mInstallLock) {
            return performDexOptInternalWithDependenciesLI(p, pkgSetting, options); //进入这里
        }
    } finally {
        Binder.restoreCallingIdentity(callingId);
    }
}
 
 
private int performDexOptInternalWithDependenciesLI(AndroidPackage p,
        @NonNull PackageSetting pkgSetting, DexoptOptions options) {
    // System server gets a special path.
    if (PLATFORM_PACKAGE_NAME.equals(p.getPackageName())) {
        return mDexManager.dexoptSystemServer(options);//android 走系统
    }
 
    // Select the dex optimizer based on the force parameter.
    // Note: The force option is rarely used (cmdline input for testing, mostly), so it's OK to
    //       allocate an object here.
    PackageDexOptimizer pdo = options.isForce()
            ? new PackageDexOptimizer.ForcedUpdatePackageDexOptimizer(mPackageDexOptimizer)
            : mPackageDexOptimizer; //如果有携带f就要强制编译，无其他逻辑
 
    // Dexopt all dependencies first. Note: we ignore the return value and march on
    // on errors.
    // Note that we are going to call performDexOpt on those libraries as many times as
    // they are referenced in packages. When we do a batch of performDexOpt (for example
    // at boot, or background job), the passed 'targetCompilerFilter' stays the same,
    // and the first package that uses the library will dexopt it. The
    // others will see that the compiled code for the library is up to date.
    Collection<SharedLibraryInfo> deps = findSharedLibraries(pkgSetting);
    final String[] instructionSets = getAppDexInstructionSets(
            AndroidPackageUtils.getPrimaryCpuAbi(p, pkgSetting),
            AndroidPackageUtils.getSecondaryCpuAbi(p, pkgSetting));
    if (!deps.isEmpty()) {
        DexoptOptions libraryOptions = new DexoptOptions(options.getPackageName(),
                options.getCompilationReason(), options.getCompilerFilter(),
                options.getSplitName(),
                options.getFlags() | DexoptOptions.DEXOPT_AS_SHARED_LIBRARY);
        for (SharedLibraryInfo info : deps) {
            AndroidPackage depPackage = null;
            PackageSetting depPackageSetting = null;
            synchronized (mLock) {
                depPackage = mPackages.get(info.getPackageName());
                depPackageSetting = mSettings.getPackageLPr(info.getPackageName());
            }
            if (depPackage != null && depPackageSetting != null) {
                // TODO: Analyze and investigate if we (should) profile libraries.
                pdo.performDexOpt(depPackage, depPackageSetting, instructionSets, //先对依赖库进行dex优化
                        getOrCreateCompilerPackageStats(depPackage),
                        mDexManager.getPackageUseInfoOrDefault(depPackage.getPackageName()),
                        libraryOptions);
            } else {
                // TODO(ngeoffray): Support dexopting system shared libraries.
            }
        }
    }
 
    return pdo.performDexOpt(p, pkgSetting, instructionSets,//进入PackageDexOptimizer#performDexOpt流程，上面已经分析过。
            getOrCreateCompilerPackageStats(p),
            mDexManager.getPackageUseInfoOrDefault(p.getPackageName()), options);
}

dexOpt & odex

一、现存问题

二、理论基础

1、ART 执行方式

2、编译选项

3、强制编译命令

4、手动执行dex2Oat

三、PKMS 代码执行逻辑

1、pkms执行逻辑

2、BackgroundDexOptService

相关文章

网友评论

延伸阅读

深度阅读

栏目导航

热点阅读