#751629 xserver-xorg-video-vmware: Conflicting declarations of function MakeAtom to cause undefined behaviour

Package:
xserver-xorg-video-vmware
Source:
xserver-xorg-video-vmware
Description:
X.Org X server -- VMware display driver
Submitter:
Michael Tautschnig
Date:
2024-08-05 09:24:03 UTC
Severity:
normal
#751629#5
Date:
2014-06-14 21:29:03 UTC
From:
To:
During a rebuild of all Debian packages in a clean sid chroot (using cowbuilder
and pbuilder) the build failed with the following error. Please note that we
use our research compiler tool-chain (using tools from the cbmc package), which
permits extended reporting on type inconsistencies at link time.

[...]
libtool: link: gcc -std=gnu99 -shared  -fPIC -DPIC  .libs/vmware_drv_la-bits2pixels.o .libs/vmware_drv_la-vmware.o .libs/vmware_drv_la-vmwarecurs.o .libs/vmware_drv_la-vmwarectrl.o .libs/vmware_drv_la-vmwarexinerama.o .libs/vmware_drv_la-vmwarevideo.o .libs/vmware_drv_la-vmwaremodes.o .libs/vmware_drv_la-vmware_bootstrap.o .libs/vmware_drv_la-vmware_common.o  -Wl,--whole-archive ../vmwgfx/.libs/libvmwgfx.a -Wl,--no-whole-archive  -ldrm -lxatracker  -O2   -Wl,-soname -Wl,vmware_drv.so -o .libs/vmware_drv.so

error: conflicting function declarations "MakeAtom"
old definition in module vmwarevideo file /usr/include/xorg/dix.h line 268
unsigned int (const char *, unsigned int, signed int)
new definition in module vmwgfx_overlay file /usr/include/xorg/dix.h line 268
unsigned long int (const char *, unsigned int, signed int)
Makefile:476: recipe for target 'vmware_drv.la' failed
make[3]: *** [vmware_drv.la] Error 64

For any platform with sizeof(unsigned int)<sizeof(unsigned long int) the
behaviour is undefined as either several bytes remain undefined (if the true
return type is unsigned int) or several bytes will be missing (if the true
return type is unsigned long int). Also, endianness will add further trouble.

The actual cause of the problem remains to be determined as both declarations
stem from the same source, yet expand in different ways.

Best,
Michael

#751629#10
Date:
2024-08-04 05:32:32 UTC
From:
To:
[Michael Tautschnig 2014-06-14]

I tried to make sense of this, but failed.  This is the declaration in
the current edition of /usr/include/xorg/dix.h:

  extern _X_EXPORT Atom MakeAtom(const char * /*string */ ,
                                 unsigned /*len */ ,
                                 Bool /*makeit */ );

I fail to figure out how the Atom type can become unsigned int in one
incarnation and unsigned long int in another.  Perhaps the problem
went away the last ten years?

#751629#15
Date:
2024-08-05 09:21:11 UTC
From:
To:
The definition of Atom depends on _XSERVER64, which is defined by xorg-server.h or dix-config.h. The driver must include one of those headers before the header which defines Atom to ensure it gets the correct definition.