Plotting in batch with no display

[Hans Georg Schaathun]

Admittedly not the strongest reason, but yet an important one,
for switching from Matlab to python/numpy/scipy/matplotlib,
is that Matlab is very cumbersome to run in batch.

Now I discover that some of the matplotlib.pyplot functions
(incl. plot and contour) insist on opening an X11 window
(just like Matlab does). I would have preferred just to create
the plot straight on file. The extra window is a nuisance on my
laptop, it is deep-felt pain if I try to run it remotely. It fails
completely if I run it without any display at all.

Oddly, the bar() function does not open a window by default.
I was very happy with that. It works exactly the way I want.
(Why isn't pyplot more consistent?)

Is there something I have missed? Is it possible to create
standard 2D plots and contour plots without a display, writing
the graphics straight into a PDF file?

I'll be grateful for any pointers and ideas,
:-- George

 

[Emile van Sebille]

On 6/4/2010 1:01 PM Hans Georg Schaathun said...

Admittedly not the strongest reason, but yet an important one,
for switching from Matlab to python/numpy/scipy/matplotlib,
is that Matlab is very cumbersome to run in batch.

Now I discover that some of the matplotlib.pyplot functions
(incl. plot and contour) insist on opening an X11 window
(just like Matlab does).

I found the same to be true when running OpenOffice in batch mode.
Ultimately, the following was key to getting things going:

at the shell:

/usr/bin/vncserver :1


then from within python:

XDISPLAY=':1'

cmd='''cd /usr/ftp/CSV && DISPLAY=%s \
/usr/bin/soffice -norecover -nologo -nodefault \
"macro:///Standard.Module1.csvToXLS(%s)"''' \
% (XDISPLAY,csvfile)

dummy = commands.getoutput(cmd)


Then, if it doesn't run to conclusion, you can vnc to :1 and see where
it's stalled out.

HTH,

Emile

 

[exarkun]

Admittedly not the strongest reason, but yet an important one,
for switching from Matlab to python/numpy/scipy/matplotlib,
is that Matlab is very cumbersome to run in batch.

Now I discover that some of the matplotlib.pyplot functions
(incl. plot and contour) insist on opening an X11 window
(just like Matlab does). I would have preferred just to create
the plot straight on file. The extra window is a nuisance on my
laptop, it is deep-felt pain if I try to run it remotely. It fails
completely if I run it without any display at all.

Oddly, the bar() function does not open a window by default.
I was very happy with that. It works exactly the way I want.
(Why isn't pyplot more consistent?)

Is there something I have missed? Is it possible to create
standard 2D plots and contour plots without a display, writing
the graphics straight into a PDF file?

It's possible to plot with matplotlib without a display. I'm not
surprised you didn't figure out how, though, it's not all that obvious.

Check out the matplotlib.use function. For example:

import matplotlib
matplotlib.use('agg')
import pylab
pylab.plot([1, 3, 5])
fig = file('foo.png', 'wb')
pylab.savefig(fig, format='png')
fig.close()

This should work fine without a display.

Jean-Paul

 

[asdef]

It's possible to plot with matplotlib without a display. I'm not
surprised you didn't figure out how, though, it's not all that
obvious.

http://matplotlib.sourceforge.net/faq/howto_faq.html#plotting-howto
,----
| Generate images without having a window popup
`----

 

[Hans Georg Schaathun]

It's possible to plot with matplotlib without a display. I'm not
: surprised you didn't figure out how, though, it's not all that obvious.

Thank you very much. That's a good start. Do you know of any good
documentation on how to choose and use backends? What I find
is very thin...

: Check out the matplotlib.use function. For example:
:
: import matplotlib
: matplotlib.use('agg')
: import pylab
: pylab.plot([1, 3, 5])
: fig = file('foo.png', 'wb')
: pylab.savefig(fig, format='png')
: fig.close()

Raster graphics is not good enough, I will need a backend which
does vector graphics and pdf output. AFAICS from the FAQ at
sourceforge, agg only supports raster and png. Cairo supports
vector graphics and PDF, but I cannot find any information about
how/if it deals with X11...

:-- George

 

[Tim Harig]

Raster graphics is not good enough, I will need a backend which
does vector graphics and pdf output. AFAICS from the FAQ at
sourceforge, agg only supports raster and png. Cairo supports
vector graphics and PDF, but I cannot find any information about
how/if it deals with X11...

1. I am not familiar with matlab; but, I have written python programs that
generate plots using GNUplot (which can create PDFs). There are
python modules available for working with GNUplot or you can open
it yourself under a subprocess and send it commands through a pipe.
GNUplot allows you to save files directly and does not therefore
require X.

2. If you can generate a raster file with Matlab, you can use ImageMagick
(which has python bindings or could be execed directly) or other
conversion software to convert the file to whatever format you
choose.

3. Depending on your requirements, you could generate a plot using
postscript. Postscript is easy to convert directly to PDF as
the two are closely related. As postscript is a powerful
language itself you can actually write the graphing routines in
postscript and use python to simply embed the data into the
postscript.

 

[Hans Georg Schaathun]

> Raster graphics is not good enough, I will need a backend which
: > does vector graphics and pdf output. AFAICS from the FAQ at
: > sourceforge, agg only supports raster and png. Cairo supports
: > vector graphics and PDF, but I cannot find any information about
: > how/if it deals with X11...
:
: 1. I am not familiar with matlab;

Good that my question did not concern matlab then

: but, I have written python programs that
: generate plots using GNUplot (which can create PDFs). There are
: python modules available for working with GNUplot or you can open
: it yourself under a subprocess and send it commands through a pipe.
: GNUplot allows you to save files directly and does not therefore
: require X.

That certainly is an option. I actually do have an example using
gnuplot, taken from the libsvm tools which I wanted to replace with
a native python solution After some testing, I have concluded
though, that that's not necessary. OTOH, gnuplot is not quite as
easy to learn as the pyplot API.

: 2. If you can generate a raster file with Matlab, you can use ImageMagick
: (which has python bindings or could be execed directly) or other
: conversion software to convert the file to whatever format you
: choose.

Sorry, that's no help. Firstly, when I started using python it was to
avoid using matlab... Secondly, raster graphics gives poor quality when
it is scaled. Raster graphics represented in PDF or EPS is no better
than raster graphics in PNG. I need scalable vector graphics.

: 3. Depending on your requirements, you could generate a plot using
: postscript. Postscript is easy to convert directly to PDF as
: the two are closely related. As postscript is a powerful
: language itself you can actually write the graphing routines in
: postscript and use python to simply embed the data into the
: postscript.

Sure, but the problem is to find the right matplotlib engine to do
that. We live in 2010 and Python is a new and evolving language.
It would be strange if the PS renderer is more advanced than the
PDF renderer, but if it is, I'll use it.

Ideally, I'd like the option to switch between X11 and file output
runtime, which the engine switch (matplotlib.use()) does not support.
Question is, is there an engine to do that?

:-- George

 

[Tim Harig]

> Raster graphics is not good enough, I will need a backend which
: > does vector graphics and pdf output. AFAICS from the FAQ at
: > sourceforge, agg only supports raster and png. Cairo supports
: > vector graphics and PDF, but I cannot find any information about
: > how/if it deals with X11...
:
: 1. I am not familiar with matlab;

Good that my question did not concern matlab then

Sorry for the disclaimer. Matlab seemed to be referenced quite a bit and
since I am not familiar with it, I am likely to misunderstand something
that might otherwise be obvious.

: but, I have written python programs that
: generate plots using GNUplot (which can create PDFs). There are
: python modules available for working with GNUplot or you can open
: it yourself under a subprocess and send it commands through a pipe.
: GNUplot allows you to save files directly and does not therefore
: require X.

That certainly is an option. I actually do have an example using
gnuplot, taken from the libsvm tools which I wanted to replace with
a native python solution After some testing, I have concluded
though, that that's not necessary. OTOH, gnuplot is not quite as
easy to learn as the pyplot API.

GNUplot is very easy to learn. I started using it after reading a three
page tutorial. After you get the idea of how it works, everything else is
just details that you can learn from the online help pages from within
GNUplot itself.

I have never actually used any API to access GNUPlot from python as I
already knew the commands anyway so I have always just given my own data
objects the capability to send the necessary commands through the pipe.
However, just looking at the demo code file for the API at:

http://gnuplot-py.sourceforge.net/

I looks like I could basically use it just based on my GNUplot knowledge.
Its really just a *very* transparent class encapsulation of the GNUplot
commands. (And it creates the same pipe that I use for my code on the
backside)

: 3. Depending on your requirements, you could generate a plot using
: postscript. Postscript is easy to convert directly to PDF as
: the two are closely related. As postscript is a powerful
: language itself you can actually write the graphing routines in
: postscript and use python to simply embed the data into the
: postscript.

Sure, but the problem is to find the right matplotlib engine to do
that. We live in 2010 and Python is a new and evolving language.

I just through it out there as a possible option. This is not the kind of
thing that is probably useful for most projects; but, it can be extremely
useful for some situations.

Having a viewing environment based on a Turing complete language means
that you can implement a flexible MVC view componet that the user can
use to manipulate the view of the data simply for the price of having
a Postscript viewer. This leaves your application free to focus solely on
the business logic. For comparison, think about an AJAX based web page
where the client merely receives structured data from the server (XML,
JSON, etc) and javascript on the client actually allows the user to choose
how the data is viewed.

Sending data embedded in a postscript file also means that the data is
sent, and can be independantly extracted, along with the graph image itself.

It would be strange if the PS renderer is more advanced than the
PDF renderer, but if it is, I'll use it.

I frankly don't know much about PDF other than it is a technology closely
related to Postscript. I have heard it phrased that PDFs are essentially
based on Postscript technology where the loops have been "unrolled." I
personally think of PDFs as a form of compiled Postscript where the program
is executed and the results are simply saved and there is no way to
re-execute the program using different data.

As far as the language itself, I find Postscript to be rather intriguing.
It uses a very different philosophy from Python; but, it has some really
neat features that I haven't really seen in other langauges.

Ideally, I'd like the option to switch between X11 and file output
runtime, which the engine switch (matplotlib.use()) does not support.
Question is, is there an engine to do that?

GNUplot chooses its outputs by setting the "terminal" variable. Terminal
can be set to a text terminal, a graphical terminal, or a number of
different file formats.

 

[Mark Lawrence]

> Raster graphics is not good enough, I will need a backend which
:> does vector graphics and pdf output. AFAICS from the FAQ at
:> sourceforge, agg only supports raster and png. Cairo supports
:> vector graphics and PDF, but I cannot find any information about
:> how/if it deals with X11...
:
: 1. I am not familiar with matlab;

Good that my question did not concern matlab then

: but, I have written python programs that
: generate plots using GNUplot (which can create PDFs). There are
: python modules available for working with GNUplot or you can open
: it yourself under a subprocess and send it commands through a pipe.
: GNUplot allows you to save files directly and does not therefore
: require X.

That certainly is an option. I actually do have an example using
gnuplot, taken from the libsvm tools which I wanted to replace with
a native python solution After some testing, I have concluded
though, that that's not necessary. OTOH, gnuplot is not quite as
easy to learn as the pyplot API.

I entirely agree with this comment.

: 2. If you can generate a raster file with Matlab, you can use ImageMagick
: (which has python bindings or could be execed directly) or other
: conversion software to convert the file to whatever format you
: choose.

Sorry, that's no help. Firstly, when I started using python it was to
avoid using matlab... Secondly, raster graphics gives poor quality when
it is scaled. Raster graphics represented in PDF or EPS is no better
than raster graphics in PNG. I need scalable vector graphics.

: 3. Depending on your requirements, you could generate a plot using
: postscript. Postscript is easy to convert directly to PDF as
: the two are closely related. As postscript is a powerful
: language itself you can actually write the graphing routines in
: postscript and use python to simply embed the data into the
: postscript.

Sure, but the problem is to find the right matplotlib engine to do
that. We live in 2010 and Python is a new and evolving language.
It would be strange if the PS renderer is more advanced than the
PDF renderer, but if it is, I'll use it.

Ideally, I'd like the option to switch between X11 and file output
runtime, which the engine switch (matplotlib.use()) does not support.
Question is, is there an engine to do that?

:-- George

For more specific data have you tried the matplotlib specific mailing
list at e.g. gmane.comp.python.matplotlib.general.

Apologies if this has been suggested in this thread and I've missed it.

Kindest regards.

Mark Lawrence.

 

[Giacomo Boffi]

: import matplotlib
: matplotlib.use('agg')
: import pylab
: pylab.plot([1, 3, 5])
: fig = file('foo.png', 'wb')
: pylab.savefig(fig, format='png')
: fig.close()

Raster graphics is not good enough

#ig = file('foo.png', 'wb'
pylab.savefig('foo.png', format='pdf')

that's all

 

[Giacomo Boffi]

: import matplotlib
: matplotlib.use('agg')
: import pylab
: pylab.plot([1, 3, 5])
: fig = file('foo.png', 'wb')
: pylab.savefig(fig, format='png')
: fig.close()

Raster graphics is not good enough

#ig = file('foo.png', 'wb'
pylab.savefig('foo.png', format='pdf')

that's all

either

matplotlib.use('cairo.pdf')
....
pylab.savefig('foo.pdf')