大师兄的Python源码学习笔记(三十七）: 模块的动态加载机制

大师兄的Python源码学习笔记(三十六）: 模块的动态加载机制（三）
大师兄的Python源码学习笔记(三十八）: 模块的动态加载机制（五）

三、import机制的实现

1. 找到module/package

• 这一部分实际分析了大师兄的Python源码学习笔记(三十四）: 模块的动态加载机制（一）中，PyImport_ImportModuleLevelObject函数到PyImport_GetModule函数之间发生的故事。
• 回到熟悉的PyImport_ImportModuleLevelObject函数：

Python\import.c

PyObject *
PyImport_ImportModuleLevelObject(PyObject *name, PyObject *globals,
                                 PyObject *locals, PyObject *fromlist,
                                 int level)
{
    _Py_IDENTIFIER(_handle_fromlist);
    PyObject *abs_name = NULL;
    PyObject *final_mod = NULL;
    PyObject *mod = NULL;
    PyObject *package = NULL;
    PyInterpreterState *interp = PyThreadState_GET()->interp;
    int has_from;
    ... ...

        mod = import_find_and_load(abs_name);
    ... ...

• 在经过一系列判断后，调用核心函数import_find_and_load：

Python\import.c

static PyObject *
import_find_and_load(PyObject *abs_name)
{
    _Py_IDENTIFIER(_find_and_load);
    PyObject *mod = NULL;
    PyInterpreterState *interp = PyThreadState_GET()->interp;
    int import_time = interp->core_config.import_time;
    static int import_level;
    static _PyTime_t accumulated;

    ... ...

    mod = _PyObject_CallMethodIdObjArgs(interp->importlib,
                                        &PyId__find_and_load, abs_name,
                                        interp->import_func, NULL);

    ... ...

    return mod;
}

• 跳转到Python并调用Python函数_find_and_load_unlocked:

Lib\importlib\_bootstrap.py

def _find_and_load_unlocked(name, import_):
    path = None
    parent = name.rpartition('.')[0]
    if parent:
        if parent not in sys.modules:
            _call_with_frames_removed(import_, parent)
        # Crazy side-effects!
        if name in sys.modules:
            return sys.modules[name]
        parent_module = sys.modules[parent]
        try:
            path = parent_module.__path__
        except AttributeError:
            msg = (_ERR_MSG + '; {!r} is not a package').format(name, parent)
            raise ModuleNotFoundError(msg, name=name) from None
    spec = _find_spec(name, path)
    if spec is None:
        raise ModuleNotFoundError(_ERR_MSG.format(name), name=name)
    else:
        module = _load_unlocked(spec)
    if parent:
        # Set the module as an attribute on its parent.
        parent_module = sys.modules[parent]
        setattr(parent_module, name.rpartition('.')[2], module)
    return module

• _find_and_load_unlocked函数会通过_find_spec函数找到需要加载的模块在哪里，然后调用_load_unlocked函数来加载模块。
• 首先仔细观察_find_spec函数：

Lib\importlib\_bootstrap.py

def _find_spec(name, path, target=None):
    """Find a module's spec."""
    meta_path = sys.meta_path
    if meta_path is None:
        # PyImport_Cleanup() is running or has been called.
        raise ImportError("sys.meta_path is None, Python is likely "
                          "shutting down")

    if not meta_path:
        _warnings.warn('sys.meta_path is empty', ImportWarning)

    # We check sys.modules here for the reload case.  While a passed-in
    # target will usually indicate a reload there is no guarantee, whereas
    # sys.modules provides one.
    is_reload = name in sys.modules
    for finder in meta_path:
        with _ImportLockContext():
            try:
                find_spec = finder.find_spec
            except AttributeError:
                spec = _find_spec_legacy(finder, name, path)
                if spec is None:
                    continue
            else:
                spec = find_spec(name, path, target)
        if spec is not None:
            # The parent import may have already imported this module.
            if not is_reload and name in sys.modules:
                module = sys.modules[name]
                try:
                    __spec__ = module.__spec__
                except AttributeError:
                    # We use the found spec since that is the one that
                    # we would have used if the parent module hadn't
                    # beaten us to the punch.
                    return spec
                else:
                    if __spec__ is None:
                        return spec
                    else:
                        return __spec__
            else:
                return spec
    else:
        return None

• _find_spec函数会遍历sys.meta_path, 对其中的每一个对象调用findspec函数。
• 查看sys.meta_path：

>>> sys.meta_path
[<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]

• Python模块按导入方式一共分为六种：

1.builtin module loader
2.frozen module loader
3.source file loader
4.sourceless file loader
5.extension module loader
6.zipimport loader

• 可以看到sys.meta_path包含的三个对象: 其中BuiltinImporter和FrozenImporter分别对应模块加载的前两种方法，而后四种加载方法都是由PathFinder实现的。
• 查看PathFinder中的find_spec方法：

Lib\importlib\_bootstrap_external.py

class PathFinder:

    """Meta path finder for sys.path and package __path__ attributes."""

    ... ...

    @classmethod
    def find_spec(cls, fullname, path=None, target=None):
        """Try to find a spec for 'fullname' on sys.path or 'path'.

        The search is based on sys.path_hooks and sys.path_importer_cache.
        """
        if path is None:
            path = sys.path
        spec = cls._get_spec(fullname, path, target)
        if spec is None:
            return None
        elif spec.loader is None:
            namespace_path = spec.submodule_search_locations
            if namespace_path:
                # We found at least one namespace path.  Return a spec which
                # can create the namespace package.
                spec.origin = None
                spec.submodule_search_locations = _NamespacePath(fullname, namespace_path, cls._get_spec)
                return spec
            else:
                return None
        else:
            return spec

    ... ...

• PathFinder.findspec会对sys.path_hooks里的每一个hook函数分别调用sys.path里的每一个路径。
• 如果某一个hook函数在指定的路径下发现了指定的包，它就会返回一个_bootstrap.ModuleSpec对象。
• 在_bootstrap.ModuleSpec中会有一个loader对象负责真正加载这个包：

Lib\importlib\_bootstrap_external.py

def _get_supported_file_loaders():
    """Returns a list of file-based module loaders.

    Each item is a tuple (loader, suffixes).
    """
    extensions = ExtensionFileLoader, _imp.extension_suffixes()
    source = SourceFileLoader, SOURCE_SUFFIXES
    bytecode = SourcelessFileLoader, BYTECODE_SUFFIXES
    return [extensions, source, bytecode]

• PathFinder返回的ModuleSpec可能的loader有以下三种:ExtensionFileLoader, SourceFileLoader, SourcelessFileLoader。
• 接着观察_load_unlocked函数：

Lib\importlib\_bootstrap.py

def _load_unlocked(spec):
    # A helper for direct use by the import system.
    if spec.loader is not None:
        # not a namespace package
        if not hasattr(spec.loader, 'exec_module'):
            return _load_backward_compatible(spec)

    module = module_from_spec(spec)
    with _installed_safely(module):
        if spec.loader is None:
            if spec.submodule_search_locations is None:
                raise ImportError('missing loader', name=spec.name)
            # A namespace package so do nothing.
        else:
            spec.loader.exec_module(module)

    # We don't ensure that the import-related module attributes get
    # set in the sys.modules replacement case.  Such modules are on
    # their own.
    return sys.modules[spec.name]

• _load_unlocked函数创建了一个空的module对象，然后将这个空moudle对象作为参数调用了具体loader的exec_module函数。
• 回到作为exec_module函数的ExtensionFileLoader类：

Lib\importlib\_bootstrap_external.py

class ExtensionFileLoader(FileLoader, _LoaderBasics):

    """Loader for extension modules.

    The constructor is designed to work with FileFinder.

    """

    ... ...

    def create_module(self, spec):
        """Create an unitialized extension module"""
        module = _bootstrap._call_with_frames_removed(
            _imp.create_dynamic, spec)
        _bootstrap._verbose_message('extension module {!r} loaded from {!r}',
                         spec.name, self.path)
        return module
    ... ...

• 在ExtensionFileLoader中调用了builtin对象_imp.create_dynamic,它的实现在C语言函数_imp_create_dynamic_impl：

Python\import.c

static PyObject *
_imp_create_dynamic_impl(PyObject *module, PyObject *spec, PyObject *file)
/*[clinic end generated code: output=83249b827a4fde77 input=c31b954f4cf4e09d]*/
{
    PyObject *mod, *name, *path;
    FILE *fp;

    ... ...

    mod = _PyImport_LoadDynamicModuleWithSpec(spec, fp);

    Py_DECREF(name);
    Py_DECREF(path);
    if (fp)
        fclose(fp);
    return mod;
}

• 最后进入_PyImport_LoadDynamicModuleWithSpec函数：

PyObject *
_PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *fp)
{
... ...

#ifdef MS_WINDOWS
    exportfunc = _PyImport_FindSharedFuncptrWindows(hook_prefix, name_buf,
                                                    path, fp);
#else
    pathbytes = PyUnicode_EncodeFSDefault(path);
    if (pathbytes == NULL)
        goto error;
    exportfunc = _PyImport_FindSharedFuncptr(hook_prefix, name_buf,
                                             PyBytes_AS_STRING(pathbytes),
                                             fp);
    Py_DECREF(pathbytes);
#endif

    ... ...

    p0 = (PyObject *(*)(void))exportfunc;

    /* Package context is needed for single-phase init */
    oldcontext = _Py_PackageContext;
    _Py_PackageContext = PyUnicode_AsUTF8(name_unicode);
    if (_Py_PackageContext == NULL) {
        _Py_PackageContext = oldcontext;
        goto error;
    }
    m = p0();
   ... ...
}

• 最终程序将调用dlopen/LoadLibrary来加载动态链接库并且执行其中的PyInit_modulename。