lkubuntu

A listing of random software, tips, tweaks, hacks, and tutorials I made for Ubuntu

Writing a python plugin API/Architecture

The first time I attempted to write a plugin API was for a C++ project. Long story short, the project didn’t have one when it was released (and never did). Since then I was always scared of writing one in any language, but since my python project (relinux) required one, I decided to try. It was actually quite a lot easier than I expected, since all that was required to do was to get a list of plugins and to load them each into a variable.

We’ll start by creating the “header” of the plugin loader

import imp
import os

PluginFolder = "./plugins"
MainModule = "__init__"

The imp module is needed for finding and importing the plugins, and the rest should be self-explanatory.
Next we’ll create a function for finding plugins

def getPlugins():
    plugins = []
    possibleplugins = os.listdir(PluginFolder)
    for i in possibleplugins:
        location = os.path.join(PluginFolder, i)
        if not os.path.isdir(location) or not MainModule + ".py" in os.listdir(location):
            continue
        info = imp.find_module(MainModule, [location])
        plugins.append({"name": i, "info": info})
    return plugins

Most of this code should be self-explanatory. It simply looks through the directories in the plugin folder, and tries to find MainModule.py in each of them. If it finds the MainModule module, it will load information about it using imp.find_module (this information is used by imp’s module loader).
All that is left now is to add the plugin loading function

def loadPlugin(plugin):
    return imp.load_module(MainModule, *plugin["info"])

This function is simple enough (the asterisk simply extends the “info” list into arguments). The module returned can be used like any module object as it’s simply a namespace.

Here is the full code of the plugin loader:

import imp
import os

PluginFolder = "./plugins"
MainModule = "__init__"

def getPlugins():
    plugins = []
    possibleplugins = os.listdir(PluginFolder)
    for i in possibleplugins:
        location = os.path.join(PluginFolder, i)
        if not os.path.isdir(location) or not MainModule + ".py" in os.listdir(location):
            continue
        info = imp.find_module(MainModule, [location])
        plugins.append({"name": i, "info": info})
    return plugins

def loadPlugin(plugin):
    return imp.load_module(MainModule, *plugin["info"])

Here is an example of using this module (assuming you named it “pluginloader”)

for i in pluginloader.getPlugins():
    print("Loading plugin " + i["name"])
    plugin = pluginloader.loadPlugin(i)
    plugin.run()

Here is a sample plugin (in ./plugins/hello/__init__.py):

def run():
    print("Hello from a plugin!")

Now, of course, this plugin API is very simple, and can easily (and should) be extended for your program’s needs.

About these ads

31 responses to “Writing a python plugin API/Architecture

  1. txwikinger October 3, 2012 at 12:12 am

    Reblogged this on txwikinger's blog.

  2. Julian Andres Klode October 3, 2012 at 10:15 am

    But you do know that folder is the wrong word, right? And encouraging mixed case naming is really not helpful in a world where almost everyone else follows PEP 8’s lowercase with underscores variant

    • lkjoel October 3, 2012 at 5:39 pm

      Yeah, that would be easy to change. This is by no means a “perfect” implementation that you should use as-is, so it should be very easy to integrate the styling changes :)

  3. Mikko Ohtamaa October 3, 2012 at 3:58 pm

    I suggest you don’t invent your own API, but use Python eggs and entry points if you wish to have third party plug-ins http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/advanced_pylons/entry_points_and_plugins.html

    • lkjoel October 3, 2012 at 5:42 pm

      That seems like it would be great for larger projects that require larger plugins, but as I said in the post, this is a very simple small-project plugin API :)

  4. yabapolido December 16, 2012 at 12:35 am

    Hi,
    Thanks for sharing! I was just looking for something like this.
    However, let me ask you some help.
    Having this code, how can I implement something like:
    – MainApp has a command prompt
    – MainApp has 3 or 4 commands available that user can type
    – Plugin 1 has 2 new commands
    – Plugin 2 has 3 new commands
    – MainApp should process those 5 new commands and whenever the user types one of them, MainApp should redirect them to the correspondent plugin and then execute something.
    Thanks in advance!
    keep up the good work

    • lkjoel December 16, 2012 at 3:10 am

      Glad to know it helps :D. Here is how I would do it (might not be the best way, but it should work): Each plugin will have a dictionary of the commands your plugin offers, to which it points to the function. Sort of like this:
      import sys

      def sayhello():
      print("Hello world!")

      def echo():
      print(input("Enter text to echo: ") if sys.version_info[0] > 2 else raw_input("Enter text to echo: "))

      commands = {
      "hello": sayhello,
      "echo": echo
      }

      So then in your main application, you would also use the same technique (heck, you could even make it so that the commands you have in MainApp would be a plugin), and then you could simply load them using a for loop, like this (assuming plugins points to the array of plugins, and command_entered is the string of the command to process):
      for i in plugins:
      if command_entered in i.commands.keys():
      i.commands[command_entered]()

      Hope this helps!

      • yabapolido December 16, 2012 at 1:13 pm

        @lkjoel, Thanks! I wasn’t excpeting a answser so soon and accurate :)
        I had that theory but couldn’t pass it to code.
        Hope you don’t mind, I’ve made a reference to this post at:

        http://stackoverflow.com/questions/13897330/python-plugins-like-minecraft-bukkit

        Thanks again, this is really what I’ve needed.

      • yabapolido December 16, 2012 at 1:26 pm

        I’m sorry, I can’t figure out what’s wrong :|
        plugins = getPlugins()
        while 1:
        foo = raw_input(“:”)
        for i in plugins:
        if foo in i.commands.keys():
        i.commands[foo]
        I get:
        AttributeError: ‘dict’ object has no attribute ‘commands’
        However, I do have the array “commands” at the plugin :(

      • yabapolido December 16, 2012 at 1:34 pm

        getPlugins() returns an array right?
        Do I have to load the plugin again so that “commands” be available ?!

      • lkjoel December 16, 2012 at 6:29 pm

        You have to first load the plugin, like this:
        plugins_meta = getPlugins()
        plugins = []
        for i in plugins_meta:
        plugins.append(loadPlugin(i))

        Then it should work (getPlugins() returns the metadata for loading it, not the actual plugin). Note that with the code you provided, you might have another error message, as plugin.commands[foo] is a function, so you would have to call it (add the () after it)

      • yabapolido December 17, 2012 at 1:32 am

        The plugins.append(loadPlugin(i)) is not working. I’ve double checked everything, having 10 plugins, the array will be filled with 10 copies of the last plugin :|
        Any idea ?!

      • lkjoel December 17, 2012 at 3:50 am

        To be honest, I’m not sure where the problem is. Why don’t you try first copying the code from the post to the place where it should be (just in case you accidentally rewrote something), and then place checks throughout the “getPlugins()” function (like “print(plugins)”). Is everything normal throughout? Also make sure that the “plugins” folder contains the proper directory structure (“plugins/pluginname/__init__.py”).

  5. yabapolido December 16, 2012 at 6:32 pm

    AH. this single line was my problem:
    plugins.append(loadPlugin(i)) :) :) :) :) :) :)
    I wasn’t doing any of this, and didn’t think it was possible at all :) Python still has obscure stuff to me.

    • yabapolido December 16, 2012 at 6:34 pm

      I don’t want to bother you, but now I get:
      AttributeError: ‘list’ object has no attribute ‘commands’ :| :|

      • lkjoel December 16, 2012 at 6:36 pm

        Are you sure that you the variable you are using to access “commands” is actually referring to the plugin? The “loadPlugin” function should return a module object (to which there should be the “commands” property, as you added it to the plugin), not a list object (is it the plugin array you were referring to?).

      • yabapolido December 16, 2012 at 6:38 pm

        I have this:

        snip…
        def loadPlugin(plugin):
        return imp.load_module(MainModule, *plugin[“info”])

        plugins_meta = getPlugins()
        xplugins = []
        for i in plugins_meta:
        xplugins.append(loadPlugin(i))

        while 1:
        foo = raw_input(“:”)
        if foo in xplugins.commands.keys():
        xplugins.commands[foo]()
        elif foo == ‘quit':
        break

      • lkjoel December 16, 2012 at 6:46 pm

        Oh, I see. Try this:
        while 1:
        foo = raw_input(":")
        command_recognized = false
        for i in xplugins:
        if foo in i.commands.keys():
        command_recognized = true
        i.commands[foo]()
        if not command_recognized:
        if foo == "quit":
        break

        This should work, as it iterates through each plugin in the list, then accesses the properties from there. Your code was trying to access properties from the list of plugins (which is not possible).

      • yabapolido December 16, 2012 at 6:46 pm

        Agh… I’m so sorry, I was missing the for loop

      • yabapolido December 17, 2012 at 12:19 am

        For one plugin it’s working, when I add the second, it duplicates the commands :(
        I have this (sorry, can’t formate as code):

        while 1:
        foo = raw_input(“:”).split()
        if len(foo) == 1:
        if (foo[0] == ‘quit’) or (foo[0] == ‘q’):
        break
        elif foo[0] == ‘help':
        print “Plugins available:”
        print disponiveis
        elif foo[0] == ‘listall':
        print “Plugins commands available:”
        print todos
        elif len(foo) >= 1:
        #print disponiveis
        if foo[0] in disponiveis:
        for i in plugins:
        #print i.commands.keys()
        if foo[1] in i.commands.keys():
        i.commands[foo[1]]()

      • yabapolido December 17, 2012 at 12:22 am

        Fixed, missed a break, however, It’s only executing the commands of the last plugin loaded :(

      • lkjoel December 17, 2012 at 12:26 am

        Could you paste the whole source code (using pastebin or pastie or something, because it would be hard to read if pasted here…)?

      • yabapolido December 17, 2012 at 12:30 am

        First of all, thanks for all the help. Here’s the link

        http://pastebin.com/Q2mHYaV7

      • lkjoel December 17, 2012 at 12:49 am

        On line 60 I see “a.commands.keys()”… is this intentional?

      • yabapolido December 17, 2012 at 12:57 am

        No, I was changing the code when copy/paste.
        Anyway. at line 32 just do a “print plugins”, the array has only 1 element :| i can’t figure out why. When it loads, it shows both of them.

      • lkjoel December 17, 2012 at 12:58 am

        I’m sorry, could you update the pastebin? Line 32 there is blank

    • yabapolido December 17, 2012 at 1:02 am

      On line 32 I’ve just added “print plugins” to debugging purposes, this is the result:

      Loading plugins…
      …luz
      …tv
      [, ]

      You see? the array has “tv” twice, however, when doing:
      “for i in plugins_meta:”
      it prints both of them, and i guess plugins.append should add both of them too, i’m I wrong?

      • yabapolido December 17, 2012 at 1:04 am

        Oops… can’t copy/paste here. it’s missing the most important stuff, the contents of plugins array.

  6. Boubakr December 24, 2012 at 4:01 pm

    It’ll be a great implementation if you used OOP, class and exception…
    any way, nice one :D

    • MiJyn December 25, 2012 at 7:08 pm

      I’m personally not a fan of those technologies, so I usually try to find alternative methods :)

  7. kev December 2, 2013 at 4:53 am

    Thanks for the post, its very informative.

    I know I am late to the party but it may be useful to point out that if you have multiple plugins then each time loadPlugin(plugin) is called the module __init__ is overridden.

    If you replace:
    imp.load_module(MainModule, *plugin[“info”])
    with
    imp.load_module(plugin[“name”], *plugin[“info”]).

    Then the modules are imported as the plugin name.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 179 other followers

%d bloggers like this: