I inserted several png images in typo3 4.2.5 using typo3 backend.

When I press the "save" button in typo3 backend, several gm processes
consume 100% of the cpus. One process per image.

It seems as if gm scales the images right for the frontend: The resolution
differs from the original resolution. The corresponding page can be found
here:
http://www.slicewise.net/index.php?id=58

But I think typo3 wants to make some thumbnails and this might lead to the
problem.

I can't see any problems in the typo3 test suite, all graphics look great.

I use the current typo3 version from the repository of debian lenny.

Hi!

Could you please try to get hold of the full command line typo3 is using, and
also make available one of the original images used as input when the problem
occurred? For how long did it hog the CPUs? Right now, it's not clear to me
whether this is a bug in GraphicsMagick, or just typo3 starting a
time-consuming task.

Regards,

Daniel.

Prior to the 1.3.8 (January 21, 2010) or 1.2.10 (January 6, 2010)
GraphicsMagick releases, there was a GraphicsMagick hang which was
noticed under TYPO3:

   * Eliminate lockup due to hanging in loop while parsing malformed
     sub-image specification (SourceForge issue 2886560).

The problem was triggered by a TYPO3 bug which has been fixed for a
long time now.

It may be that this bug report is for something new, but it is
worthwhile to check that it is not a re-occurance of an already known
scenario.

Bob

Maybe it's the same issue as reported in
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=638409 for ImageMagick.

I also saw a similar problem with a Perl script using either
GraphicsMagick or ImageMagick on a virtual machine instance. After
rebuilding graphicsmagick using the --without-openmp switch in configure
everything worked OK.

Regards,
    Slaven

Please try using the OMP_WAIT_POLICY environment variable as described
at
"http://gcc.gnu.org/onlinedocs/libgomp/OMP_005fWAIT_005fPOLICY.html#OMP_005fWAIT_005fPOLICY".

See if setting OMP_WAIT_POLICY=PASSIVE in the server environment makes
much of the CPU issue go away.

Note that setting OMP_NUM_THREADS to a small number will help if the
server uses fork() and so server processes are competing with each
other.  For point of reference, Flickr (which uses the forking model)
sets OMP_NUM_THREADS=2 and also uses GOMP_CPU_AFFINITY to bind work
processes to certain pre-determined CPUs.

GOMP does seem to be more wasteful of CPU than some other OpenMP
implementations.  It is pretty typical for an OpenMP implementation to
try a busy spin-lock for a short time (e.g. 10ms) before reverting to
an OS mutex.  This may even include while an idle OpenMP thread is
waiting for a task assignment.  If the thread is always in the busy
spin-lock then it will consume 100% CPU.  Even if it is in the busy
spin-lock for a shorter time, it will be indicated to have consumed
100% CPU since Linux evaluates CPU consumption based on if the thread
was scheduled during that scheduler tick.  It seems that when
OMP_WAIT_POLICY is set to ACTIVE GOMP uses busy spin-lock while if it
is set to PASSIVE it goes straight to OS mutex.  Otherwise it likely
uses a mix.

Check out the difference in the reported user (actually user + system)
time consumed and the computed "iter/s cpu" for these various
settings:

Minimal operations (memory-bandwidth bound):

% env gm benchmark -duration 5 convert -size 4000x3000 tile:model.pnm -noop null:
Results: 24 threads 251 iter 65.43s user 5.01s total 50.100 iter/s (3.836 iter/s cpu)

% env OMP_WAIT_POLICY=ACTIVE gm benchmark -duration 5 convert -size 4000x3000 tile:model.pnm -noop null:
Results: 24 threads 180 iter 116.18s user 5.01s total 35.928 iter/s (1.549 iter/s cpu)

% env OMP_WAIT_POLICY=PASSIVE gm benchmark -duration 5 convert -size 4000x3000 tile:model.pnm -noop null:
Results: 24 threads 232 iter 39.06s user 5.01s total 46.307 iter/s (5.940 iter/s cpu)

Rotate image by 90 degrees (lots of thread contention):

% gm benchmark -duration 5 convert -size 4000x3000 tile:model.pnm -rotate 90 null:
Results: 24 threads 71 iter 54.86s user 5.05s total 14.059 iter/s (1.294 iter/s cpu)

% env OMP_WAIT_POLICY=ACTIVE gm benchmark -duration 5 convert -size 4000x3000 tile:model.pnm -rotate 90 null:
Results: 24 threads 55 iter 116.00s user 5.07s total 10.848 iter/s (0.474 iter/s cpu)

% env OMP_WAIT_POLICY=PASSIVE gm benchmark -duration 5 convert -size 4000x3000 tile:model.pnm -rotate 90 null:
Results: 24 threads 71 iter 45.29s user 5.07s total 14.004 iter/s (1.568 iter/s cpu)

Image resize:

% gm benchmark -duration 5 convert -size 4000x3000 tile:model.pnm -resize 50% null:
Results: 24 threads 38 iter 100.31s user 5.09s total 7.466 iter/s (0.379 iter/s cpu)

% env OMP_WAIT_POLICY=ACTIVE gm benchmark -duration 5 convert -size 4000x3000 tile:model.pnm -resize 50% null:
Results: 24 threads 39 iter 118.29s user 5.01s total 7.784 iter/s (0.330 iter/s cpu)

% env OMP_WAIT_POLICY=PASSIVE gm benchmark -duration 5 convert -size 4000x3000 tile:model.pnm -resize 50% null:
Results: 24 threads 37 iter 88.86s user 5.08s total 7.283 iter/s (0.416 iter/s cpu)

Bob

Here is a bash script which may be used to run multiple copies of
GraphicsMagick in parallel working on the same algorithm.  The
specified total number of iterations is subdivided across a specified
number of GM processes.  Any GM subcommand may be specified. This may
be useful in order to understand Linux VM loading and the best OpenMP
options to use in order to avoid wasting precious shared VM CPU cyles
while still achieving good throughput.

The subcommand used for the example run output is one which normally
consumes 100% of available CPU.

Bob

% cat ./benchparallel.sh
#!/bin/bash

# Run gm in parallel with enough processes to subdivide the work to
# achieve the requested number of total iterations.  Useful to
# determine OpenMP parameters in a forking-server environment.
#
# See http://gcc.gnu.org/onlinedocs/libgomp/index.html#toc_Environment-Variables
# for libgomp tunables.
#
# Usage
#   ./benchparallel.sh total_iter nprocs [subcommand] [input_image] [output_image]
#
# Typical usage:
#   ./benchparallel.sh 12 3 '-gaussian 0x1'
typeset -i i=0
typeset -i total_iter=$1
typeset -i nprocs=$2
subcommand=${3:-'-noop'}
input_image=${4:-'-size 4000x3000 tile:logo:'}
output_image=${5:-'null:'}
typeset -i iter_per_proc
duration=5
let iter_per_proc=total_iter/nprocs
command="gm benchmark -iterations ${iter_per_proc} convert ${input_image} ${subcommand} ${output_image}"
echo "Executing ${nprocs}x: ${command} ..."
time (
   while ((i < nprocs))
   do
     eval "${command}" &
     let i=i+1
   done
   # Wait for processing to be done
   wait
)


% ./benchparallel.sh 24 4 '-gaussian 0x1'
Executing 4x: gm benchmark -iterations 6 convert -size 4000x3000 tile:logo: -gaussian 0x1 null: ...
Results: 24 threads 6 iter 59.09s user 10.04s total 0.598 iter/s (0.102 iter/s cpu)
Results: 24 threads 6 iter 58.96s user 10.07s total 0.596 iter/s (0.102 iter/s cpu)
Results: 24 threads 6 iter 59.06s user 10.08s total 0.595 iter/s (0.102 iter/s cpu)
Results: 24 threads 6 iter 58.76s user 10.09s total 0.595 iter/s (0.102 iter/s cpu)

real    0m10.100s
user    3m51.462s
sys     0m4.468s

% env OMP_NUM_THREADS=6 ./benchparallel.sh 24 4 '-gaussian 0x1'
Executing 4x: gm benchmark -iterations 6 convert -size 4000x3000 tile:logo: -gaussian 0x1 null: ...
Results: 6 threads 6 iter 57.05s user 9.81s total 0.612 iter/s (0.105 iter/s cpu)
Results: 6 threads 6 iter 56.76s user 9.83s total 0.610 iter/s (0.106 iter/s cpu)
Results: 6 threads 6 iter 56.80s user 9.83s total 0.610 iter/s (0.106 iter/s cpu)
Results: 6 threads 6 iter 57.11s user 9.99s total 0.601 iter/s (0.105 iter/s cpu)

real    0m9.990s
user    3m43.742s
sys     0m4.024s

% env OMP_NUM_THREADS=4 ./benchparallel.sh 24 4 '-gaussian 0x1'
Executing 4x: gm benchmark -iterations 6 convert -size 4000x3000 tile:logo: -gaussian 0x1 null: ...
Results: 4 threads 6 iter 54.94s user 13.88s total 0.432 iter/s (0.109 iter/s cpu)
Results: 4 threads 6 iter 55.77s user 14.18s total 0.423 iter/s (0.108 iter/s cpu)
Results: 4 threads 6 iter 55.94s user 14.23s total 0.422 iter/s (0.107 iter/s cpu)
Results: 4 threads 6 iter 55.95s user 14.23s total 0.422 iter/s (0.107 iter/s cpu)

real    0m14.242s
user    3m39.142s
sys     0m3.492s

  env OMP_NUM_THREADS=4 OMP_WAIT_POLICY=PASSIVE ./benchparallel.sh 24 4 '-gaussian 0x1'
Executing 4x: gm benchmark -iterations 6 convert -size 4000x3000 tile:logo: -gaussian 0x1 null: ...
Results: 4 threads 6 iter 56.54s user 14.36s total 0.418 iter/s (0.106 iter/s cpu)
Results: 4 threads 6 iter 56.04s user 14.46s total 0.415 iter/s (0.107 iter/s cpu)
Results: 4 threads 6 iter 56.27s user 14.55s total 0.412 iter/s (0.107 iter/s cpu)
Results: 4 threads 6 iter 56.43s user 14.55s total 0.412 iter/s (0.106 iter/s cpu)

real    0m14.560s
user    3m41.766s
sys     0m3.564s

% env OMP_NUM_THREADS=24 OMP_WAIT_POLICY=PASSIVE ./benchparallel.sh 24 1 '-gaussian 0x1'
Executing 1x: gm benchmark -iterations 24 convert -size 4000x3000 tile:logo: -gaussian 0x1 null: ...
Results: 24 threads 24 iter 237.87s user 11.30s total 2.124 iter/s (0.101 iter/s cpu)

real    0m11.306s
user    3m52.047s
sys     0m5.840s