zfs commands return EINVAL

[openzfs]

zfs commands return EINVAL

[seriv]

I want to add some additional bits about zpool behavior.

I have zfs and pulled yesterday from github:

[seriv@gauntlet zfs]$ git log | head
commit 8c3ab23f4be92a4e55f07f8c1bb467b851ed3e54
Author: Brian Behlendorf 
Date:   Fri Nov 5 12:29:58 2010 -0700

    Add lustre zpios-test workload
...
[seriv@gauntlet zfs]$ cd ../spl
[seriv@gauntlet spl]$ git log | head
commit 1e18307b6109a5b2491cd519a312b870b2e7522b
Author: Brian Behlendorf 
Date:   Mon Nov 8 21:32:47 2010 -0800

    Fix incorrect krw_type_t type
...

I have destroyed/exported all zpools I have.

[seriv@gauntlet ~]$ sudo zpool list
no pools available
[seriv@gauntlet ~]$ sudo dd if=/dev/zero of=/dev/sda3 bs=8192 count=8192
8192+0 records in
8192+0 records out
67108864 bytes (67 MB) copied, 1.25259 s, 53.6 MB/s

Then created new zpool and have the same problems:

[seriv@gauntlet ~]$ sudo zpool create newtest /dev/sda3
[seriv@gauntlet ~]$ sudo zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
-         -      -      -      -      -       -  -
[seriv@gauntlet ~]$ sudo zpool status
  pool: newtest
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        newtest     ONLINE       0     0     0
          sda3      ONLINE       0     0     0

errors: No known data errors

And strace zpool status gave me the same ioctl EINVAL.

[behlendorf]

Interesting, what distribution are you using? Also is it 32-bit or 64-bit? I haven't seen anything like this in my testing, but I could image it might come up if say your using a 32-bit user space and 64-bit kernel and the ioctl numbers aren't mapped the same.

[seriv]

It's Fedora 14 64-bit, I think I have some 32-bit libs and applications like acroread, but most are 64-bit:

[seriv@gauntlet ~]$ cat /etc/redhat-release 
Fedora release 14 (Laughlin)
[seriv@gauntlet ~]$ uname -a
Linux gauntlet.localdomain 2.6.35.6-48.fc14.x86_64 #1 SMP Fri Oct 22 15:36:08 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
[seriv@gauntlet ~]$ file `which zpool`
/usr/sbin/zpool: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, not stripped
[seriv@gauntlet ~]$ rpm -qf `which zpool`
zfs-0.5.2-1.x86_64

[behlendorf]

Thanks for the info, when I get a little time I'll setup FC14 in a new VM and see if I can reproduce this. Although, since this is an older bug presumably you saw this with FC13 too?

[seriv]

I have not tried it on Fedora-13 because it is already upgraded to Fedora-14. But tested on Fedora-12 and have the same problems. One additional thing was with 'sudo make check'. It reported problem:

[seriv@parkheights zfs]$ sudo make check
Making check in etc
make[1]: Entering directory `/home/seriv/work/zfs/etc'
make[1]: Nothing to be done for `check'.
make[1]: Leaving directory `/home/seriv/work/zfs/etc'
Making check in man
make[1]: Entering directory `/home/seriv/work/zfs/man'
Making check in man8
make[2]: Entering directory `/home/seriv/work/zfs/man/man8'
make[2]: Nothing to be done for `check'.
make[2]: Leaving directory `/home/seriv/work/zfs/man/man8'
make[2]: Entering directory `/home/seriv/work/zfs/man'
make[2]: Nothing to be done for `check-am'.
make[2]: Leaving directory `/home/seriv/work/zfs/man'
make[1]: Leaving directory `/home/seriv/work/zfs/man'
Making check in scripts
make[1]: Entering directory `/home/seriv/work/zfs/scripts'
Making check in zpool-config
make[2]: Entering directory `/home/seriv/work/zfs/scripts/zpool-config'
make[2]: Nothing to be done for `check'.
make[2]: Leaving directory `/home/seriv/work/zfs/scripts/zpool-config'
Making check in zpool-layout
make[2]: Entering directory `/home/seriv/work/zfs/scripts/zpool-layout'
make[2]: Nothing to be done for `check'.
make[2]: Leaving directory `/home/seriv/work/zfs/scripts/zpool-layout'
Making check in zpios-test
make[2]: Entering directory `/home/seriv/work/zfs/scripts/zpios-test'
make[2]: Nothing to be done for `check'.
make[2]: Leaving directory `/home/seriv/work/zfs/scripts/zpios-test'
Making check in zpios-profile
make[2]: Entering directory `/home/seriv/work/zfs/scripts/zpios-profile'
make[2]: Nothing to be done for `check'.
make[2]: Leaving directory `/home/seriv/work/zfs/scripts/zpios-profile'
make[2]: Entering directory `/home/seriv/work/zfs/scripts'
make[2]: Nothing to be done for `check-am'.
make[2]: Leaving directory `/home/seriv/work/zfs/scripts'

==================================== ZTEST ====================================

5 vdevs, 7 datasets, 23 threads, 300 seconds...
props:
    autoreplace: 1
Setting dataset ztest/ds_3 to sync always
Setting dataset ztest/ds_4 to sync always
Setting dataset ztest/temp_0 to sync always
Setting dataset ztest/ds_5 to sync always
Setting dataset ztest/temp_19 to sync always
Setting dataset ztest/temp_10 to sync always

Pass   1,  SIGKILL,   0 ENOSPC,  6.5% of  238M used,   8% done,    4m37s to go
Pass   2,  SIGKILL,   0 ENOSPC,  6.2% of  238M used,   9% done,    4m32s to go
Setting dataset ztest/temp_19 to sync always
Setting dataset ztest/temp_7 to sync always
Setting dataset ztest/temp_2 to sync always
Pass   3, Complete,   0 ENOSPC,  5.9% of  508M used,  41% done,    2m57s to go
Pass   4, Complete,   0 ENOSPC,  4.4% of  814M used,  85% done,      44s to go
Pass   5,  SIGKILL,   0 ENOSPC,  4.4% of  814M used,  87% done,      38s to go
Pass   6, Complete,   0 ENOSPC,  4.4% of  850M used, 100% done,       0s to go
3 killed, 3 completed, 50% kill rate


=================================== ZCONFIG ===================================

Skipping test 10 which requires the scsi_debug  module and the /usr/bin/lsscsi utility
1    persistent zpool.cache             Pass
2    scan disks for pools to import     Pass
3    zpool import/export device         Pass
4    zpool insmod/rmmod device          Pass
5    zvol+ext3 volume                   Pass
6    zvol+ext2 snapshot                 cannot create snapshot 'tank/fish@pristine': out of space
Fail (8)
[seriv@parkheights zfs]$ df -h | grep tank/fish
/dev/tank/fish1       388M  2.3M  365M   1% /tmp/fish1
[seriv@parkheights zfs]$ sudo  cmd/zfs/zfs list
NAME        USED  AVAIL  REFER  MOUNTPOINT
tank        413M  52.8M  31.4K  /tank
tank/fish   413M   452M  13.9M  -
[seriv@parkheights zfs]$ sudo  cmd/zfs/zfs list -t snapshot
no datasets available

But this problem was not "stable", when I have run "sudo strace -f cmd/zfs/zfs snapshot tank/fish@pristine", it worked without reproducing the problem.

And zpool list still can't get zpool names,

[seriv@parkheights zfs]$ sudo dd if=/dev/zero bs=8192 of=/dev/sdd3 count=8192
8192+0 records in
8192+0 records out
67108864 bytes (67 MB) copied, 1.00835 s, 66.6 MB/s
[seriv@parkheights zfs]$ sudo zpool list
no pools available
[seriv@parkheights zfs]$ sudo zpool create test /dev/sdd3
[seriv@parkheights zfs]$ sudo zpool status
  pool: test
 state: ONLINE
 scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    test        ONLINE       0     0     0
      sdd3      ONLINE       0     0     0

errors: No known data errors
[seriv@parkheights zfs]$ sudo zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
-         -      -      -      -      -       -  -
[seriv@parkheights zfs]$ sudo zfs create test/mytest
cannot create 'test/mytest': no such parent 'test'

[seriv@parkheights zfs]$ uname -a
Linux parkheights.dyndns.org 2.6.32.23-170.fc12.x86_64 #1 SMP Mon Sep 27 17:23:59 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
[seriv@parkheights zfs]$ cat /etc/redhat-release 
Fedora release 12 (Constantine)

[behlendorf]

It seems to work fine for me on FC12 and FC14, there must be something slightly different about your setup. OK, let's focus on the 'zpool list' problem which your originally reported. The other issues are likely transient false positives in the test cases.

Your original bug report shows the that 'zpool list' command was failing because ioctl 0x5a28 was failing with EINVAL. Can you please verify that it's now failing on 0x5a0a, the numbering was changed slightly in newer releases. This is ZFS_IOC_POOL_GET_PROPS which is what 'zpool list' calls to populate that table. It should look something like this in the strace.

ioctl(3, 0x5a2a, 0x7fff8dc15a60)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a2a, 0x7fff8dc15a60)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a2a, 0x7fff8dc15a60)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a2a, 0x7fff8dc15a60)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a2a, 0x7fff8dc15a60)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a2a, 0x7fff8dc15a60)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a2a, 0x7fff8dc15a60)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a2a, 0x7fff8dc15a60)        = -1 EINVAL (Invalid argument)
write(1, "-         -      -      -      -"..., 51-         -      -      -      -      -       -  -
) = 51

If that's true then the issue is somewhere in the zfs_ioc_pool_get_props() call path. Unfortunately, EINVAL is a fairly common error code which is returned in multiple locations, we need to determine exactly which check is failing and why.

[seriv]

Strange, on Fedora-14 I have

[seriv@gauntlet zfs]$ sudo strace zpool list 2>&1 | grep -C5 EINVAL
open("/dev/zfs", O_RDWR)                = 3
open("/proc/mounts", O_RDONLY)          = 4
open("/etc/dfs/sharetab", O_RDONLY)     = -1 ENOENT (No such file or directory)
ioctl(3, 0x5a04, 0x7fff7e55ed80)        = 0
ioctl(3, 0x5a05, 0x7fff7e55ed40)        = 0
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc0b57ee000
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7e55dd00)        = -1 EINVAL (Invalid argument)
close(3)                                = 0
close(4)                                = 0
write(1, "NAME   SIZE  ALLOC   FREE    CAP"..., 108NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
-         -      -      -      -      -       -  -
) = 108
[seriv@gauntlet zfs]$ rpm -qV zfs
[seriv@gauntlet zfs]$ which zpool
/usr/sbin/zpool
[seriv@gauntlet zfs]$ rpm -qf `which zpool`
zfs-0.5.2-1.x86_64
[seriv@gauntlet zfs]$ rpm -qV zfs-0.5.2-1.x86_64

Zfs there is fresh git clone of commit 8c3ab23.

And on Fedora-12:

[seriv@parkheights ~]$ sudo strace zpool list 2>&1 | grep -C5 EINVAL
open("/dev/zfs", O_RDWR)                = 3
open("/proc/mounts", O_RDONLY)          = 4
open("/etc/dfs/sharetab", O_RDONLY)     = -1 ENOENT (No such file or directory)
ioctl(3, 0x5a04, 0x7fff7d0cc6c0)        = 0
ioctl(3, 0x5a05, 0x7fff7d0cc680)        = 0
ioctl(3, 0x5a28, 0x7fff7d0cb640)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7d0cb640)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7d0cb640)        = -1 EINVAL (Invalid argument)
fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb895918000
ioctl(3, 0x5a28, 0x7fff7d0cb630)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7d0cb630)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7d0cb630)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7d0cb630)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7d0cb630)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7d0cb630)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7d0cb630)        = -1 EINVAL (Invalid argument)
ioctl(3, 0x5a28, 0x7fff7d0cb630)        = -1 EINVAL (Invalid argument)
close(3)                                = 0
close(4)                                = 0
write(1, "NAME   SIZE  ALLOC   FREE    CAP"..., 108NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
-         -      -      -      -      -       -  -
) = 108

and there zfs is clone of commit b04cffc

[behlendorf]

Are you absolutely sure your not accidentally using an old version of zpool which was installed on your system? That would cause exactly this issue and explain why your seeing this issue and I'm not. For the updated code ZFS_IOC_OBJ_TO_PATH is 0x5a28 and ZFS_IOC_POOL_GET_PROPS is 0x5a2a (include/sys/zfs/zfs.h:719), I would absolutely expect it to return EINVAL is your using an old version of zpool. That would also explain why things like 'zpool status' still work, those ioctl number did not shift. I've tried hard to ensure these numbers don't change but it couldn't be helped.

[seriv]

You are right. when I run zpool in zfs checkout directory, I do not have the problem, and I think the problem may be closed.

[seriv@gauntlet zfs]$ sudo  ./cmd/zpool/zpool list 
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
newtest  56.5G  82.5K  56.5G     0%  1.00x  ONLINE  -

Now I need to understand how can "rpm -qV zfs' to show nothing, "rpm -qa zfs" show version 0.5.2-1 and I have old versions of zpool in the rpm packages.
But this is totally different problem.

[behlendorf]

Ahh good, I'm glad that was it. I'm closing this bug.