Dear Maintainer,

I bootstrapped a Debian base system via "debootstrap --arch i386 wheezy ./newfiles http://http.debian.net/debian/" and put it on a Galileo board. On the Galileo board there new Intel Quark IA processor - which is basically a 486 with some instructions extensions from Pentium.
If I boot the existing Galileo kernel with the bootstrapped fileset, many applications crash with a segfault in pthread.
To reporduce, follow the instruction on https://communities.intel.com/message/220080
Here are ways to reproduce consistently - many other apps show the same behavior
1. Boot the system, create a new user (non root), connect to the board via ssh - the sshd will crash with a segfault in pthread
2. Do a 'apt-get install cowsay' - at the end, apt-get will crash with a segfault in pthread

sshd[2519]: segfault at b7173107 ip b714f07b sp bf97ea94 error ffff0007 in libpthread-2.13.so[b714a000+15000]

incorrect behavior: segfault - applications stop working
expected behavior: no crash

uname -a = Linux galileo 3.8.7-yocto-standard #1 Wed Jan 15 00:21:32 CET 2014 i586 GNU/Linux
dpkg -s libc6 = 2.13-38

The default toolchain (and thus libc) in Debian has been targetting
i586 for quite a while now.  If this CPU doesn't provide *all* the
i586 instructions, I'd be pretty surprised if anything worked.

... Adam

The default toolchain (and thus libc) in Debian has been targetting
i586 for quite a while now.  If this CPU doesn't provide *all* the
i586 instructions, I'd be pretty surprised if anything worked.

... Adam

I was first trending in the same direction, but this is a different issue.
I compiled code with instructions that aren't working on Quark. The
issue you get then is an 'invalid instruction exception', not a
segfault.
If you use gcc and the '-march=i586' option, the code that is
generated is fine for Quark.

The problem happens in __nptl_setxid, at address 0x507b:

00005060 <__nptl_setxid>:
    5060:       55                      push   %ebp
    5061:       31 c0                   xor    %eax,%eax
    5063:       89 e5                   mov    %esp,%ebp
    5065:       b9 01 00 00 00          mov    $0x1,%ecx
    506a:       57                      push   %edi
    506b:       56                      push   %esi
    506c:       53                      push   %ebx
    506d:       83 ec 14                sub    $0x14,%esp
    5070:       e8 fb f3 ff ff          call   4470 <__i686.get_pc_thunk.bx>
    5075:       81 c3 7f 0f 01 00       add    $0x10f7f,%ebx
=>  507b:       f0 0f b1 8b 94 21 00    lock cmpxchg %ecx,0x2194(%ebx)
    5082:       00
    5083:       0f 85 b6 17 00 00       jne    683f <_L_lock_743>
    5089:       8b 45 08                mov    0x8(%ebp),%eax
    508c:       8b b3 38 01 00 00       mov    0x138(%ebx),%esi

Despite the name __i686.get_pc_thunk.bx is fine on this CPU (it
actually has been rename to __x86.get_pc_thunk.bx on recent GCC
versions), as it is only get the PC through the stack with a movl
instruction:

00004470 <__i686.get_pc_thunk.bx>:
    4470:       8b 1c 24                mov    (%esp),%ebx
    4473:       c3                      ret

So the question is if the "lock cmpxchg" instruction is behaving
correctly on the Intel Quark. This should be supported according
to the developer's manual.

It might be difficult to investigate more without access to such a CPU.

I have access to a couple of these boards. I'll be glad to run some tests if needed.


                                 Stuart

Stuart R. Anderson                               anderson@netsweng.com
Network & Software Engineering                   http://www.netsweng.com/
1024D/37A79149:                                  0791 D3B8 9A4C 2CDC A31F
                                                  BD03 0A62 E534 37A7 9149

The test would be to run one of the affected program under gdb, add a
breakpoint on __nptl_setxid, and understand why the memory access fail.

With gdb attached, the crash does not happen. I've got some more things
to try though.


                                 Stuart

Stuart R. Anderson                               anderson@netsweng.com
Network & Software Engineering                   http://www.netsweng.com/
1024D/37A79149:                                  0791 D3B8 9A4C 2CDC A31F
                                                  BD03 0A62 E534 37A7 9149

I think that it's a race condition as it doesn't happen every time

Hi,

I patched out the lock instruction via hexedit and the segfaults are gone!

before:

=>  507b:       f0 0f b1 8b 94 21 00    lock cmpxchg %ecx,0x2194(%ebx)

after:

    507b:       90                      nop
    507c:       0f b1 8b 94 21 00 00    cmpxchg %ecx,0x2194(%ebx)


Might be a bug in the CPU itself?


With kind regards
Thomas

Intel knows about this, see the errata (page 15):
http://downloadmirror.intel.com/23197/eng/Quark_SW_RelNotes_330232_001.pdf


With kind regards
Thomas

Thanks for digging on this issue, that indeed explains a lot.

It's probably something we can fix in glibc using the workaround
proposed by Intel, that said "lock" is a quite common instruction as
soon as you have multithreaded code, so there are probably hundred of
packages to fix (which seems to confirmed by codesearch.debian.net).

The best would be to get the problem fixed at the hardware level, either
through a new chip revision or a new microcode.

While it shows that the problem is at the CPU level, it's not really a
fix, as the bus is not locked anymore, so it might results in issues in
multithreaded solution.

The correct solution would be to apply the solution from Intel, that is
adding a nop before every instruction with the lock prefix. This means
rebuilding the code.

cause of the bug and to get Debian running somehow on the Galileo-Board.

To my understanding the lock prefix only makes sense on multi-core
systems, so the impatient (galileo users) should be safe by removing the
prefix at every occurence until a proper fix is implemented?

About that: /proc/cpuinfo doesn't include a "microcode" entry, so I
wouldn't bet on a quick fix by Intel. I don't know if this feature is
really missing, though.

With kind regards
Thomas

For Galileo users yes, for Edison ones, no.

Let's wait a bit to see what happens on the Intel side before trying to
workaround that at the package level, especially having a more precise
description of which instruction sequence causes the issue in order to
not patch every instruction with the lock prefix if not needed.

FYI, http://en.wikipedia.org/wiki/Pentium_F00F_bug

Hi,

FWIW: I've got a Galileo board and am running into similar issuse (while
running CentOS 5 on it ;))

Exact same error at the exact same location; it's not the instruction
that is not supported , a very small piece of test code that does
  lock cmpxchgl  %edx, (%eax)

works just fine - and this instruction is used left and right in the
vmlinux kernel itself anyway.
So it must be the memory offset that is triggering error ...

JJK / Jan Just Keijser

This instruction might work in some cases, but not in all cases.

No it's not. It is a CPU bug, as stated by Intel in their release notes
(p 15), which only happens in "specific circumstances".

http://downloadmirror.intel.com/23197/eng/Quark_SW_RelNotes_330232_001.pdf

Hi,
OK I hadn't thought of reading the Quark release notes ;)

I did find a workaround, however: if I configure glibc with the extra
parameter
   -with-cpu=i386

then the assembly code generated becomes
   __nptl_setxid:
.LFB75:
         .loc 1 1046 0
.LVL87:
         pushl   %ebp
.LCFI27:
         movl    %esp, %ebp
.LCFI28:
         pushl   %edi
.LCFI29:
         pushl   %esi
.LCFI30:
         pushl   %ebx
.LCFI31:
         subl    $8, %esp
.LCFI32:
.LBB79:
         .loc 1 1049 0
         xorl    %eax, %eax
         movl    $1, %ecx
#APP
         lock;cmpxchgl %ecx, stack_cache_lock
         jnz _L_lock_698
         .subsection 1
         .type _L_lock_698,@function

and the segfault does not occur - I can now happily run a 'stock' centos
5 sshd on my galileo board. Here's my setup:
- yocto kernel 3.8.7 (from BSP 0.7.5)
- clanton image based on centos5 i386 image, using stock rpms
- modified /etc/rc.d/rc.sysinit script (don't try to mount /proc/bus/usb)
- modified /lib/libpthread.so.0 symlink to custom libpthread.so

(I'm sure I've missed a few other things ;)

share and enjoy,

JJK / Jan Just Keijser

See the new "-momit-lock-prefix" option added to binutils (gas) in
order to solve this issue:

https://sourceware.org/ml/binutils/2014-08/msg00093.html

control: tag -1 + wontfix

It might solve this issue for Intel Quark, but it would break for any
multicore processors. This is not something acceptable.

At least it confirms the issue is really on the CPU side and not the
libc side, so I am tagging this bug as wontfix.

Aurélien you are correct - some logic is required to use the
-momit-lock-prefix flag for Quark and not for all other CPUs. This is
possible in embedded, cross-compilation environments which build
everything from source (e.g. Yocto) but more difficult elsewhere.

2014-10-21 11:25 GMT+01:00 Aurelien Jarno <aurelien@aurel32.net>:

What version of glibc are you compiling? Can you offer some more
assistance as to how you compiled it? Are you cross-compiling or do have
a working set of tools on your galileo to compile natively?

I am attempting to compile from my Ubuntu 14.04 (64-bit) machine with

    ../glibc-2.20/configure -with-cpu=i386

And am receiving

    checking sysdep dirs... configure: error: The i386 subspecies of
    x86_64 is not supported.

Thanks,
David

On Fri, 09 May 2014 00:04:30 +0200 Jan Just Keijser <janjust@nikhef.nl>  wrote:

 > I did find a workaround, however: if I configure glibc with the extra
 > parameter
 > -with-cpu=i386

Hi all,

I have done some investigation here and it seems that the CPU microcode
cannot be updated on these platforms, which brings us back to fixing the
issue in software. ( I am part of the Intel BSP team for Quark platforms ).

In order to support Quark/Galileo the most straight forward approach might
be to detect the platform at runtime in glibc and insert the NOP before
LOCK only on Quark. This would involve an if statement somewhere along the
line to either check within __nptl_setxid or have a version of __nptl_setxid
specific for Quark.

Make sense?

Ray K

Dear Customer,

Please check the attachment for your item delivery details!

FedEx

-----BEGIN PGP PUBLIC KEY BLOCK-----

TF8Z9NR/G3q1dg6YsbD/tDcdHf/TCcxtgV9JPDEKRrS4Dej0CmJZxwRtOZQbN/Qwopwuw7cghJ/L
6ss8QJqwM+LhTA9PMfOiIbLHfX282neDtDmEGSwoHfCJ85UH0m1d1oSRxvxWCID4C6OxlCnSZhIH
EtCX19iOO+u3url1CsbwLbOpLYhbZh0RK3AJf1WbDa3i/7Ojn9rfvDc3XHgMszqAyLEXToqwNt8b
SzFO0gcufx8gGFRhZ9k3cAPkN6L4ia7Xexc9Ysfo8X57/DPiD1mEn48CLlaywssbmVQZ7+Js4YFb
9sczj55wtOOcGijO7oPXZy6c+YWVWo+GKIaaAqZIYaxjryoi+0eJEnB5AYQif4Cf24B6WKyV2mVO
KirdL0r5XCYvIRpHKRocMBZB2R6zp/YU70dzfVZIbOQJGFm7KPmkplQmN+YJMDwnJ6WS8Pd81cj1
yS7uCGqaGEj1OFx2k5EZPLgLH6Yf7+CGm2fZTR+5+tTbsQj4UOFq2kCktJNA+Nfrzf/pHU8JUlpY
ByGAdiLbRPQOM6of7IasFheWh1QOtLucHSibt+jUgiyoMZtnDj7ZgJnEvI6jwTkvl87XjwTIORje
5y7iYdpQsFSXFTBgj5fflCSvd9cmLdHGRImKb8+g83V7DWj2hpTQ+rRDyjwu1PWhMuw9O0ccefF1
89FgW/RpL5KVahCAoPrYyFonrah2/12VybHxQJoDBwDWN+Aj0iBsp6YDYEHjnnTkyw0PBP1RUcXy
DFg0S8v1L9+69/76+2l7vQvb/F2SfW8VbSwkus8QfMU8skv9NH+ekEl0kCulHyIKvLzcJIHqZvPk
4oPM2x16wpWSiXViTNHO6nhDAUkzDoDyIIk/Sbkp6ttKZ8e72LIvLLMVJ8lyzzueXuYvq+8GyzqD
qm2BellsS1RZSHxaFlqIl7TaByjLI+Blg3YArKR36yrS+7j+BUEeViszVspSvH0FgxT6skkPIMUr
7zOdJYqSjgqEAeHhAh7N7b4oto6O/hA0Kuuj9d+F0uzLikZ6ECq00E/WL6OjASKxyTW4JixamzFD
bnMFKRhbN4ZohN8jXGL1/mbiy3sHJ3AA82yvpOAeww5bBLZzRToMU/9ZdN/kI3cCt+laBYwV/fx4
1QxgTrsu6pACgHPbY8pQeXHB3kMdVk5LTor25a+Vz2u2oIXUpFIKdtwHbyIFMkSRq5fR2J37eryH
AfX549nH9jErjLDNryCVpnZmwss5gdWLeMwHFSexP6CSQS1xouFXZw5vboif9m7ehGSllB0Vv94N
BHfHGK+ycPBCcQrMi29k2QuJyTGK73GG1jjMwuj5ng==

-----END PGP PUBLIC KEY BLOCK-----

Good day my good friend,

Let me start by introducing myself. I am  James Musa, an accounts
officer with Bank of Africa here in Burkina Faso West Africa.

I am writing you this letter based on the latest development at my
bank whichI will like to bring to your personal edification.
($9million) transfer claim into your bank account.

Pleaded,I need your help here, do reply for more details on how we are
going to proceed if you are intrested in this grateful oppurtunity.

Get back to me at my private email address at, mrjames.musa@gmail.com


Thanks,
Mr James Musa,
+22665473407.

for all lovers of t-shirt ideas to offer or to wear on Valentine's Day

click on the amazon link below

https://amzn.to/2CdX642



best regards

for all lovers of t-shirt ideas to offer or to wear on Valentine's Day

click on the amazon link below

https://amzn.to/2CdX642

 Hälsningar!

Mitt namn är barrister Fatma Karatas, jag är en juridisk utövare som
arbetar inom alla områden av familjens jurisdiktion. Jag kontaktade dig för
att hjälpa mig med anspråket och överföringen av pengar ($6 450 000) som
lämnats kvar av min bortgångna klient som miste livet med sin familj. Jag
fick ett mandat från hans bank att förse hans närmaste anhöriga innan hans
medel konfiskeras. Det var därför jag kontaktade dig i detta ärende på
grund av att du har samma efternamn som min bortgångne klient. Han är också
infödd i ditt land och det finns ingen registrerad arvinge i hans
kontofiler hos banken. Om du är intresserad vänligen svara snarast så att
jag kan ge dig all nödvändig information för din bättre förståelse.

Med vänlig hälsning!

Advokat. Fatma Karatas Esq,
Istanbul, Turkiet,
Postnummer: 34000.

If I understand correctly, then Ray's libx1000 library[1] provides a way to
work around this in software.  It uses some LD_PRELOAD magic, and from what I
remember, it's worth being careful when using that approach.

I opened an RFP[2] for libx1000 earlier this year, and took another look at the
Debian packaging metadata in the codebase today, resulting in a few suggested
edits as a pull request on GitHub - cc'ing you to notify you about that, Ray.
I'm unsure whether some of the proposed postinst steps are required, and will
ask you about those upstream too.

[1] - http://ashroe.eu/x1000/2016/10/21/fixing-lock-prefix-on-x1000.html

[2] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037070

Hi James,

Thanks for this - you are kind to look at this issue.

It's been a long time since Intel manufactured the X1000 / Quark, I am not
sure how many are left in the wild.
Do you think this is something that Debian might want to package and ship?

Thanks,

Ray K

You're welcome - I enjoyed learning a bit about the Quark hardware,
and the esoteric lock bug.  A shame I didn't learn about it five years
ago I suppose, but there we are.
passerby who attempts to make progress on bugs that interest to me
(possibly to the annoyance of actual DMs/DDs), so my opinion is
somewhat external (i.e.: take with a grain of salt).  It's entirely
possible that the maintenance for an additional package wouldn't be
worthwhile -- and in general, 32-bit x86 support in Debian does tend
to be dwindling.  Basically: someone has to be motivated about it
enough to be responsible for the package.

On the other hand: it seemed to me based on a quick look that much of
the packaging work is already in place, and that this package would be
opt-in for anyone who wanted to use it.  The typical use case would be
people preparing root filesystems on removable/replicable storage for
later (re)attachment to Quark systems, I'd guess.  Even so: the
LD_PRELOAD and GRUB commandline stuff does make me a bit wary -
generally we shouldn't make any unnecessary or unexpected
modifications to the target system, because those should be the
responsibility of the sysadmin and not of the maintainer.

Do you know whether Intel shipped many Quark units?  I see that
manufacturing stopped in Y2019, which isn't too long ago, but I don't
know much about how widely-adopted they were.  It was the
energy-efficiency focus of them that gathered my interest in the first
place, FWIW.

I spent a not insignificant amount of time devising this solution, to get
"Debian Support"
After a few false starts and missteps, eventually I came up with LibX1000
which was a pretty effective fix IMHO.

When I started sharing it around with the Debian & Kernel folks - the
response was pretty clear.
"This is a mess, Intel should just fix the bug ... " - which honestly in
retrospect was the right thing to do.


Thrilled that you reached out about it.

[SNIP]


There were a few micro[processors,controllers] shipped under Quark.
My memory is that the X1000 didn't last long beyond 2017.
I remember seeing stacks of them (Galileo Boards) sitting gathering dust in
Frys and Maplin.
So I couldn't say how many are in the wild being used.

It was the
Boards like the Up-board (https://up-board.org/) and its successors really
filled this gap more effectively

Yep; frustrating though it can be, pushing back to figure out the
origins of bugs and resolve them there is likely the way to free up
enough developer and maintainer time, and to improve hardware and
software quality enough, to reach Reliable Technology Utopia.. should
be any day now :)

I've been thinking more about how to improve the chances that the
package could be accepted into Debian -- my suggestion would be to
rebuild it and upload it to the mentors[1] portal, where it can
(hopefully) receive review.  I've considered uploading it myself, but
I don't have hardware to test the results on, so I'd be navigating
without a way to test the results.  From personal experience
attempting packaging: the mentoring guidelines and making sure to run
lintian checks are both worthwhile.

Even so there'd be no guarantee of review or upload acceptance -- and
unfortunately the same test-hardware limitation probably applies to
most reviewers -- so I don't know whether it'd be worth your time, but
in my mind it's possible that someone attempts to install Debian on an
X1000 platform in future, learns of this bug, and then in a
hypothetical future _might_ find libx1000 in the archive, and then be
grateful for the availalble fix.

(after re-reading your blog-post, I think that there could technically
still be rare cases where the bug appears despite the package being
installed -- the mention of 98% of cases handled -- but even so, a
mostly-usable system compared to a completely useless one seems like a
big improvement)

[1] - https://mentors.debian.net/

Followup-For: Bug #738575
X-Debbugs-Cc: raykinsella78@gmail.com

I now have an X1000 Quark board to test this on (thanks, Ray), and am hoping
to find some time over the next week or two to try that out in combination with
the libx1000.git source-and-packaging repo.

Funds donated to you. Reply for claim

Funds donated to you. Reply for claim

You have been granted 1.5m. Reply for quick claims

DISCLAIMER:
This e-mail communication and any attachments may be privileged and confidential to Government Of Karnataka., and are intended only for the use of the recipients named above If you are not the addressee you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions

You have been granted 1.5m. Reply for quick claims

DISCLAIMER:
This e-mail communication and any attachments may be privileged and confidential to Government Of Karnataka., and are intended only for the use of the recipients named above If you are not the addressee you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return e-mail. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions

Hello

I write to remind you about my last email to you which you have not replied
to, I look forward to hearing from you soon.

Louisa

Hello

I write to remind you about my last email to you which you have not replied
to, I look forward to hearing from you soon.

Louisa