#857043 nfs-common: Can nfsstat be made to show timeouts per mount point or globally for the machine?

Package:
nfs-common
Source:
nfs-utils
Description:
NFS support files common to client and server
Submitter:
Petter Reinholdtsen
Date:
2024-09-14 19:09:01 UTC
Severity:
normal
Tags:
#857043#5
Date:
2017-03-07 13:36:20 UTC
From:
To:
I am looking for a way to detect if a machine is currently experiencing
NFS hangs or not, and after reading the kernel source code I noticed the
timeouts nfs statistics value that would show this.  I further came across
<URL: http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html >
which show how to get the value printed by running 'nfsstat -c'.  But on
Debian, there is no timeouts value printed by 'nfsstat -c'.

Am I doing it wrong?  This is what I get from nfsstat

Client rpc stats:
calls      retrans    authrefrsh
110415905   23636      110416360

Client nfs v3:
null         getattr      setattr      lookup       access       readlink
0         0% 60575279 54% 453762    0% 16817036 15% 14175296 12% 125       0%
read         write        create       mkdir        symlink      mknod
4207868   3% 8249979   7% 169957    0% 3626      0% 822       0% 80        0%
remove       rmdir        rename       link         readdir      readdirplus
428449    0% 3313      0% 464045    0% 289131    0% 2924687   2% 1646905   1%
fsstat       fsinfo       pathconf     commit
5921      0% 2         0% 1         0% 0         0%

I was hoping for something looking more like this:

Client rpc:
Connection oriented:
calls    badcalls  badxids  timeouts newcreds  badverfs   timers
1595799  1511      59       297      0         0          0
cantconn nomem     interrupts
1198      0         7
Connectionless:
calls    badcalls  retrans  badxids  timeouts  newcreds   badverfs
80785    3135      25029    193      9543      0          0
timers   nomem     cantsend
17399    0         0

Client nfs:
calls    badcalls  clgets   cltoomany
1640097  3112      1640097  0
Version 2: (46366 calls)
null     getattr   setattr  root     lookup     readlink  read
0 0%     6589 14%  2202 4%  0 0%     11506 24%  0 0%      7654 16%
wrcache  write     create   remove   rename     link      symlink
0 0%     13297 28% 1081 2%  0 0%     0 0%       0 0%      0 0%
mkdir    rmdir     readdir  statfs
24 0%    0 0%      906 1%   3107 6%
Version 3: (1585571 calls)
null    getattr    setattr  lookup     access     readlink  read
0 0%    508406 32% 10209 0% 263441 16% 400845 25% 3065 0%  117959 7%
write    create     mkdir    symlink    mknod    remove   rmdir
69201 4% 7615 0%    42 0%    16 0%      0 0%     7875 0%  51 0%
rename   link       readdir  readdir+   fsstat   fsinfo   pathconf
929 0%   597 0%     3986 0%  185145 11% 942 0%   300 0%   583 0%
commit
4364 0%

Client nfs_acl:
Version 2: (3105 calls)
null       getacl     setacl     getattr    access
0 0%       0 0%       0 0%       3105 100%  0 0%
Version 3: (5055 calls)
null       getacl     setacl
0 0%       5055 100%  0 0%


I can find a per-process NFS timeouts count in /proc/$$/mountstats:

device nfsserver:/nfs/server/path mounted on /nfs/server/path with fstype nfs statvers=1.1
        opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.11.12.13,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
        age:    7691364
        caps:   caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
        sec:    flavor=1,pseudoflavor=1
        events: 60575385 723194904 1026044 35107133 16087435 8158089 752137466 69862882 35157 3887894 44422751 108980223 4740836 410819 15329462 293279 52346959 13445026 0 51266 8993532 1217 0 0 0 0 0
        bytes:  164164451145 214028657512 0 0 40640451492 180817114394 11641662 44422751
        RPC iostats version: 1.0  p/v: 100003/3 (nfs)
        xprt:   tcp 925 1 6810 0 2 110441641 110416726 109 2645913361036 0 248 53868797 22200775
        per-op statistics
                NULL: 0 0 0 0 0 0 0 0
             GETATTR: 60575379 60575381 0 9544619508 6784438976 452061 76505876 78129998
             SETATTR: 453778 453779 0 90066680 65344032 63712 590276 674384
              LOOKUP: 16817043 16817043 0 3314596236 3964783208 56762 34903770 35341541
              ACCESS: 14175369 14183675 5 2301023152 1701043560 1708922 4820888 7083181
            READLINK: 125 125 0 20472 18620 0 1112 1118
                READ: 4207868 4207869 0 714491244 41184781156 89800 22545589 22729177
               WRITE: 8250269 8265635 22 183002770728 1319488588 178252491 50489943 230625045
              CREATE: 169959 169959 0 37678972 46226800 861 1029589 1038039
               MKDIR: 3626 3626 0 762440 979232 26 23616 23867
             SYMLINK: 822 822 0 212004 223456 5 5109 5157
               MKNOD: 80 80 0 20148 21760 0 299 304
              REMOVE: 428449 428449 0 79497296 61696224 3302 2706552 2737673
               RMDIR: 3313 3313 0 634516 477072 22 5724 5940
              RENAME: 464046 464046 0 129453232 120651960 7046 5882116 5908093
                LINK: 289131 289131 0 72770368 67078392 2199 2564894 2585412
             READDIR: 2924687 2924687 0 514995436 13933369380 10350 3170214 3277591
         READDIRPLUS: 1646905 1646905 0 297559972 6868670432 84713 14240605 14381440
              FSSTAT: 5921 5921 0 974272 994728 50 8681 9039
              FSINFO: 2 2 0 232 328 0 1 1
            PATHCONF: 1 1 0 116 140 0 0 0
              COMMIT: 0 0 0 0 0 0 0 0


The third number in the per-op statistics lines is the timeout count per
operation.  But I am looking for the total for the machine / mount
point, not the per process count.
#857043#10
Date:
2017-03-09 13:27:34 UTC
From:
To:
[Petter Reinholdtsen]

I was mistaken.  This is not per process, it is shared between everyone
seing the same mount points.
#857043#15
Date:
2018-12-15 09:34:00 UTC
From:
To:
Hello Pere

does

 # /usr/sbin/mountstats /path/to/mount

do what you are looking for?
the code does seem to have some reporting of timeouts
if there are retries occurring.

Kind regards
Vince
#857043#22
Date:
2024-09-14 07:49:53 UTC
From:
To:
close 857043
thanks

I guess if this is still of interest then it should be reported to nfs-utils
upstream so that it can flow downstream.
#857043#27
Date:
2024-09-14 10:13:58 UTC
From:
To:
Debugging NFS is still interesting to the few NFS system administrators
running into problems, and Linux is still not doing as good job as
Solaris did with handling NFS servers disappearing and reappearing. :)
#857043#32
Date:
2024-09-14 10:30:39 UTC
From:
To:
Hi Petter,

I do not disagree :)

But, btw, does the answer which Vincent gave you helped?

If not then maybe I have missunderstood, if you disagree on closing
feel free to reopen it as well. What though is missing needs to be
handled upstream "first".

Regards,
Salvatore
#857043#37
Date:
2024-09-14 11:14:00 UTC
From:
To:
[Salvatore Bonaccorso]

That approach is useful, but it do not provide a relible way to detect
NFS hangs.

I do not know how to interact with upstream for this issue, but would
too love for the issue to be handled upstream.