plux is the dynamic code loading framework used in LocalStack.
Plux builds a higher-level plugin mechanism around Python's entry point mechanism.
It provides tools to load plugins from entry points at run time, and to discover entry points from plugins at build time (so you don't have to declare entry points statically in your
PluginSpec: describes a
Plugin. Each plugin has a namespace, a unique name in that namespace, and a
PluginFactory(something that creates
Pluginthe spec is describing. In the simplest case, that can just be the Plugin's class).
Plugin: an object that exposes a
loadmethod. Note that it does not function as a domain object (it does not hold the plugins lifecycle state, like initialized, loaded, etc..., or other metadata of the Plugin)
PluginFinder: finds plugins, either at build time (by scanning the modules using
setuptools) or at run time (reading entrypoints of the distribution using stevedore)
PluginManager: manages the run time lifecycle of a Plugin, which has three states:
PluginSpecinstance was created
PluginSpecwas successfully invoked
loadmethod of the
Pluginwas successfully invoked
At run time, a
PluginManager uses a
PluginFinder that in turn uses stevedore to scan the available entrypoints for things that look like a
PluginManager.load(name: str) or
PluginManager.load_all(), plugins within the namespace that are discoverable in entrypoints can be loaded.
If an error occurs at any state of the lifecycle, the
PluginManager informs the
PluginLifecycleListener about it, but continues operating.
To build a source distribution and a wheel of your code with your plugins as entrypoints, simply run
python setup.py plugins sdist bdist_wheel.
How it works:
For discovering plugins at build time, plux provides a custom setuptools command
plugins, invoked via
python setup.py plugins.
The command uses a special
PluginFinder that collects from the codebase anything that can be interpreted as a
PluginSpec, and creates from it a plugin index file
plux.json, that is placed into the
.egg-info distribution metadata directory.
When a setuptools command is used to create the distribution (e.g.,
python setup.py sdist/bdist_wheel/...), plux finds the
plux.json plugin index and extends automatically the list of entry points (collected into
plux.json file becomes a part of the distribution, s.t., the plugins do not have to be discovered every time your distribution is installed elsewhere.
To build something using the plugin framework, you will first want to introduce a Plugin that does something when it is loaded.
And then, at runtime, you need a component that uses the
PluginManager to get those plugins.
This is the way we went with
LocalstackCliPlugin. Every plugin class (e.g.,
ProCliPlugin) is essentially a singleton.
This is easy, as the classes are discoverable as plugins.
Simply create a Plugin class with a name and namespace and it will be discovered by the build time
class CliPlugin(Plugin): namespace = "my.plugins.cli"
def load(self, cli): self.attach(cli) def attach(self, cli): raise NotImplementedError
class MyCliPlugin(CliPlugin): name = "my"
def attach(self, cli): # ... attach commands to cli object
now we need a
PluginManager (which has a generic type) to load the plugins for us:
```python cli = # ... needs to come from somewhere
manager: PluginManager[CliPlugin] = PluginManager("my.plugins.cli", load_args=(cli,))
plugins: List[CliPlugin] = manager.load_all()
When you have lots of plugins that are structured in a similar way, we may not want to create a separate Plugin class
for each plugin. Instead we want to use the same
Plugin class to do the same thing, but use several instances of it.
PluginFactory, and the fact that
PluginSpec instances defined at module level are discoverable (inpired
by pluggy), can be used to achieve that.
def __init__(self, service_name): self.service_name = service_name self.service = None def should_load(self): return self.service_name in config.SERVICES def load(self): module = importlib.import_module("localstack.services.%s" % self.service_name) # suppose we define a convention that each service module has a Service class, like moto's `Backend` self.service = module.Service()
def service_plugin_factory(name) -> PluginFactory: def create(): return ServicePlugin(name)
s3 = PluginSpec("localstack.plugins.services", "s3", service_plugin_factory("s3"))
dynamodb = PluginSpec("localstack.plugins.services", "dynamodb", service_plugin_factory("dynamodb"))
Then we could use the
PluginManager to build a Supervisor
class Supervisor: manager: PluginManager[ServicePlugin]
def start(self, service_name): plugin = manager.load(service_name) service = plugin.service service.start()
@plugin decorator, you can expose functions as plugins. They will be wrapped by the framework
FunctionPlugin instances, which satisfy both the contract of a Plugin, and that of the function.
```python from plugin import plugin
@plugin(namespace="localstack.configurators") def configure_logging(runtime): logging.basicConfig(level=runtime.config.loglevel)
@plugin(namespace="localstack.configurators") def configure_somethingelse(runtime): # do other stuff with the runtime object pass ```
With a PluginManager via
load_all, you receive the
FunctionPlugin instances, that you can call like the functions
runtime = LocalstackRuntime()
for configurator in PluginManager("localstack.configurators").load_all(): configurator(runtime) ```
If you are building a python distribution that exposes plugins discovered by plux, you need to configure your projects build system so other dependencies creates the
entry_points.txt file when installing your distribution.
pyproject.toml template this involves adding the
```toml [build-system] requires = ['setuptools', 'wheel', 'plux>=1.3.1'] build-backend = "setuptools.build_meta"
pip install plux
Create the virtual environment, install dependencies, and run tests
make venv make test
Run the code formatter
Upload the pypi package using twine
Trying to package localstack for Nixpkgs. Wanted to pull in tests from github to ensure that plux works, however, it's hard to know which commit was used for a particular release.
Sometimes it can be interesting to not need plux at install time, which can be facilitated by adding the entrypoints directly in the setup.py. It would be great if plux could provide a method similar to
setup.py plugins, which just creates or modifies an existing setup.py file by adding the located plugin entrypoints.
Currently the log output of the
plugins command is a bit of a mess. There's no real good way of seeing the warnings that some files weren't importable and scannable for plugins. A nicely formatted report at the end of the command would be useful, containing
It would also be useful to configure the loglevel and capture all the output created by simply importing code, similar to pytests'