May 16, 2015

Running Python script within Qt application

IronPython is a way to expose .NET objects to python interpreter (IronPython to be exact) embedded into an executable.  The problem with IronPython is the requirement for .NET: whether using the MSFT .NET framework or the mono, the WPF GUI development is time consuming and expensive, more than small projects can afford.  Qt on the other hand seems to keep simple things simple, and still scale.  But there is not a lot of concrete examples of embedding Python into a Qt GUI.  Many of the discussions are about creating a Qt GUI from Python, which I am NOT interested in.

PythonQt

I ran across PythonQt in this article.  Unlike PyQt and Qt Jambi, PythonQt is NOT designed to provide support for developers writing standalone applications. Instead, it provides facilities to embed a Python interpreter and focuses on making it easy to expose parts of the application to Python.

Scripts often need to do more than just process data, make connections, and call functions. For example, it is usually necessary for scripts to be able to create new objects of certain types to supply to the application.

To meet this need, PythonQt contains a Python module named PythonQt which you can use to access constructors and static members of all known objects. This includes the QVariant types and the Qt namespace.

Here are some example uses of the PythonQt module:

    from PythonQt import *
    print Qt.AlignLeft# Access enum values of the Qt namespace.
    print QDate.currentDate()# Access a static QDate method.
    a = QSize(1,2)# Construct a QSize object

PythonQt requires Qt version >= 4.6.1 and Python >= 2.6 or 3.3.  Ubuntu 14.04.2 LTS already has Python 3.4, as you can check with this command:

$ dpkg-query -L python3.4 3.4.0-2ubuntu1
$ ls -lgh /usr/bin/python3

Ubuntu 14.04 also has Python 2.7 right along Python 3.

Getting PythonQt

PythonQt comes in source, so I had to build it.  Download the source from http://sourceforge.net/projects/pythonqt/files/.  I downloaded the 3.0 zip package, which contains the top level qmake file PythonQt.pro, which points at the projects in folders generator, src, extensions, tests, and examples.

The build scripts currently set to use Python 2.6. You may need to tweak the build/python.prf file to set the correct Python includes and libs.  Make sure that you use the same compiler as the one that your Python distribution is built with. If you want to use another compiler, you will need to build Python yourself, using your compiler.  On my Ubuntu system, Qt5 community version downloaded from qt.io was built with gcc 4.6.1.  On the other hand, the default compiler on Ubuntu 14.04 LTS is 4.8.2.

$ gcc --version
gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2

On Linux, you need to install a Python-dev package.  For Python 3.4 (what Ubuntu comes with), the command is:

$ sudo apt-get install python3.4-dev

which also gives me the following dependent packages: libexpat1-dev libpython3.4-dev.

Edit preference files

The preference files are in build/ folder.  In common.prf, I specify the release config only.

CONFIG += release

This preference file also determines the Qt version from the qmake being used, and the output format (release/debug); appending "staticlib" will emit a static library, for example.

build/python.prf points to the python version.  To use 3.4, I just had to change to:

win32:PYTHON_VERSION=34
unix:PYTHON_VERSION=3.4

The test for whether that python version exists is the python<ver>-config utility.  While building the extensions, I realized that linking against PythonQt fails because QtCreator uses a different build directory than the source; more precisely, the linked library path is set when qmake runs at first, by this line in build/PythonQt.prf:

unix::LIBS += -L$$PWD/../lib -lPythonQt$${DEBUG_EXT}

So I modified all occurrences of BuildDirectory in PythonQt.pro.user:

<value type="QString" key="ProjectExplorer.BuildConfiguration.BuildDirectory">/home/henry/Dorking/Qt/PythonQt3.0</value>

Build in QtCreator

After the build (ignore the warnings), the examples and tests are in lib/ folder (along with the library itself).  I ran a simple example by putting the LD_LIBRARY_PATH to the lib/ folder, like this example:

~/Dorking/Qt/PythonQt3.0$ LD_LIBRARY_PATH=./lib lib/PyGettingStarted

Using the same method, I ran the unit tests successfully;

~/Dorking/Qt/PythonQt3.0$ LD_LIBRARY_PATH=./lib lib/PythonQtTest

Time now to start writing my own Qt5 GUI.

[Optional] generate wrappers for my embedded Qt build

Since Qt is highly configurable, the wrappers that come with the released PythonQt is possibly inappropriate, and a new wrapper should be generated.  In the generator/ folder of the PythonQt code, build_all.txt specifies the Qt modules that PythonQt should interface with.  For example, I modified my build_all.txt like this:

<typesystem>
  <load-typesystem name="typesystem_core.xml" generate="yes" />
  <load-typesystem name="typesystem_gui.xml" generate="yes" />
  <load-typesystem name="typesystem_sql.xml" generate="yes" />
  <load-typesystem name="typesystem_opengl.xml" generate="yes" />
  <load-typesystem name="typesystem_svg.xml" generate="yes" />
  <load-typesystem name="typesystem_network.xml" generate="yes" />
  <load-typesystem name="typesystem_xml.xml" generate="yes" />
  <load-typesystem name="typesystem_webkit.xml" generate="yes" />
  <load-typesystem name="typesystem_xmlpatterns.xml" generate="yes" />
  <load-typesystem name="typesystem_uitools.xml" generate="yes" />
  <load-typesystem name="typesystem_multimedia.xml" generate="yes" />
</typesystem>

When run with the environment variable QTDIR pointing to the root of the Qt source folder (which includes the include/ folder), the pythonqt_generator executable will look for that file, and emit the wrappers in the generated_cpp/ folder above the generator/ folder, which the PythonQt build looks for first when building.

My Qt5 GUI

As an instrument engineer, I need a low-level way to monitor what the instrument is doing.  Since it is difficult to monitor ALL aspects of even a moderately complex instrument in 1 panel, either an MDI or a stacked panel (such as a tabbed panel) is necessary to manage the different views.  For now, a simple place holder for TIFF image (often used as the raw image from the camera, for machine vision processing) display suffices.  In the main window ctor, I code the placeholder widgets like this:

ui->setupUi(this);
setWindowTitle(tr("Instrument Control and Scripting"));
mainIcon = new QIcon(":/res/ICS-icon.png"); setWindowIcon(*mainIcon);
welcomeLabel = new QLabel();

static QPixmap icstif(":/res/ICS-blue.tif");
welcomeLabel->setPixmap(icstif);

mainHrow = new QHBoxLayout();
centralWidget()->setLayout(mainHrow);//central layout is the hrow

mainTab = new QTabWidget();
mainTab->addTab(welcomeLabel, "Image");
mainHrow->addWidget(mainTab);
mainTab->setSizePolicy(QSizePolicy::Fixed, QSizePolicy::Fixed); mainTab->resize(768, 728);

On the right hand side of the GUI, I want to interact with the instrument through Python.  The most obvious usage of PythonQt is as a Python shell within a Qt GUI, for which the examples/PyScriptingConsole/ folder is the best starting point, even though it is not documented.  As a child of the QTextEdit, it can be the top level widget, or embedded as a child widget of another--as can be seen in PythonQtScriptingConsole.h:

class PYTHONQT_EXPORT PythonQtScriptingConsole : public QTextEdit
{
  Q_OBJECT

public:
  PythonQtScriptingConsole(QWidget* parent, const PythonQtObjectPtr& context, Qt::WindowFlags i = 0);
...

I want the application to have 2 main interfaces: textbox and button based control--perhaps managed in a tabbed widget--on the left, and a python shell on the right.  To pull in the PythonQtScriptingConsole, I added the following lines to the .pro file:

PYTHONQT_ROOT = ../lib/PythonQt3.0
include ( $$PYTHONQT_ROOT/build/common.prf )
include ( $$PYTHONQT_ROOT/build/PythonQt.prf )
include ( $$PYTHONQT_ROOT/build/PythonQt_QtAll.prf )

The Python shell is inserted to the right hand side of the horizontal layout (mainHrow above) like this:

#include "PythonQt.h"
#include "PythonQt_QtAll.h"
#include "gui/PythonQtScriptingConsole.h"
...

Gui::Gui(QWidget *parent) : QMainWindow(parent), ui(new Ui::Gui)
{
...
    PythonQt::init(//PythonQt::IgnoreSiteModule |
            PythonQt::RedirectStdOut);
    PythonQt_QtAll::init();
    mainContext = PythonQt::self()->getMainModule();
    shell = new PythonQtScriptingConsole(NULL, mainContext);
    shell->resize(1280-768, 728);
    mainHrow->addWidget(shell);

The result is this GUI, which shows the welcome TIFF image I made up in MSFT PPT:
As you can see, the code suggest works, and I can import the Python3 modules (because I did NOT specify PythonQt::IgnoreSiteModule in PythonQt::init).  Note that the system wide Python module folders are already in sys.path.  To get modules that do NOT come with Python3 (such as the pyserial module I tried to import in the above screenshot), PIP is the recommended way to go.  To install pip on a system, run get-pip.py as root, as in this example:

$ sudo python3 get-pip.py
Downloading pip-6.1.1-py2.py3-none-any.whl (1.1MB)
    100% |████████████████████████████████| 1.1MB 679kB/s 
Collecting setuptools
  Downloading setuptools-15.2-py2.py3-none-any.whl (501kB)
    100% |████████████████████████████████| 503kB 1.4MB/s 
Installing collected packages: pip, setuptools
Successfully installed pip-6.1.1 setuptools-15.2

Then I can install whatever modules that support pip as root again:

$ sudo pip install pyserial

And then use the module in the python shell above, like this example:

py> import serial
py> dir(serial)

Installing python packages manually on an embedded target

The pip is convenient on a networked computer.  If an embedded target lacks networking, I need to manually download the package and run the setup file to install the files on the target.  Here is an example for the hexdump package.  After extracting the zip file and moving the folder to the target's root folder, this is what the package looked like:

# ls
PKG-INFO     __main__.py  hexdump.py   setup.py
README.txt   build        hexfile.bin

After giving executable permissions to the python files, I ran the setup script:

# ./setup.py  install
running install
running build
running build_py
creating build
creating build/lib
copying hexdump.py -> build/lib
running install_lib
copying build/lib/hexdump.py -> /usr/lib/python3.4/site-packages
byte-compiling /usr/lib/python3.4/site-packages/hexdump.py to hexdump.cpython-34.pyc
running install_data
copying hexfile.bin -> /usr/lib/python3.4/site-packages/
running install_egg_info
Writing /usr/lib/python3.4/site-packages/hexdump-3.2-py3.4.egg-info

You can see that the files wind up in the correct site-packages.

Exposing the lower level SW to PythonQt

Suppose you wanted a class Instrument exposed to Python.  An example is getting and setting the names.  In PythonQt, it is done through signals and slots mechanism.  Here is api.h:

#include <QObject>
class InstrumentWrapper : public QObject {
  Q_OBJECT

public Q_SLOTS:
  class Instrument* new_Instrument(const QString& first, const QString& last);
  void delete_Instrument(class Instrument* o);
  QString firstName(class Instrument* o);
  QString lastName(class Instrument* o);
  void setFirstName(class Instrument* o, const QString& name);
  void setLastName(class Instrument* o, const QString& name); 
};

And an implementation of this API is in api.cpp:


#include "PythonQt.h"
#include "PythonQtCppWrapperFactory.h"
#include <QObject>

class Instrument {
public:
  Instrument() {}
  Instrument(const QString& first, const QString& last) {
    _firstName = first; _lastName = last; }
  QString _firstName;
  QString _lastName;
};

#include "api.h"

class Instrument* InstrumentWrapper::new_Instrument(
        const QString& first, const QString& last) {
    return new Instrument(first, last);
}
void InstrumentWrapper::delete_Instrument(Instrument* o) { delete o; }

QString InstrumentWrapper::firstName(class Instrument* o) {

    return o->_firstName;
}
QString InstrumentWrapper::lastName(class Instrument* o) {
    return o->_lastName; 
}
void InstrumentWrapper::setFirstName(class Instrument* o,
                                     const QString& name) {
    o->_firstName = name;
}
void InstrumentWrapper::setLastName(class Instrument* o,
                                    const QString& name) {
    o->_lastName = name;
}

After the C++ side is initialized, register the wrapper to PythonQt:

#include "PythonQt.h" 
#include "api.h"

int main() {
...
    PythonQt::self()->registerCPPClass("Instrument", "",
                                       "instrument",//package within PythonQt                                        
            PythonQtCreateObject<InstrumentWrapper>);
}

Finally, use it in PythonQt shell:

from PythonQt.instrument import *

inst = Instrument("John","Doe") # create a new object
print (inst)

# print the methods available
print (dir(inst))
inst.setFirstName("Mike")
inst.setLastName("Michels") 

print inst.firstName() + " " + inst.lastName())

Debugging a python script remotely

Pydev remote debugging seems the better way to go.

Cross compiling PythonQt for embedded

So far, I found 2 problems with porting PythonQt to an embedded target running Qt5:

  1. The generated wrappers that comes with PythonQt3.0 pulls in modules that are too heavy for embedded.  There should be a way to remove unwanted modules in the build_all.txt and regenerate the wrappers, but I have not figured out how to do this yet. 
  2. LONG_BIT is defined to 0 in pyport.h for some reason, and trips a C preprocessor error.  Size of long is defined to 8 (as it should be for modern gcc compiler), so LONG_BIT should be defined to 64.  I changed #error to #warning to work around the problem and see what happens.