pmem/pmdk-convert

Conversion tool for pmdk pools

pmdk-convert's Issues

Pmdk-convert succeeds when the input file is empty

Pmdk-convert succeeds when the input file is empty, although pmdk-convert requires confirmation of operation.

Existing files/dirs while building are not cleaned

When building pmdk-convert in the same folder it does not clean old data/files (at least sub-directory nvml-1.5) and even if there's a change in there, it won't appear and may cause errors (like it did around 1.5.1-rc4, when we've changed badblock source files)

Pointless flushing

There are big warnings that -convert is not crash-safe: the user has to back up the data beforehand, and restore it in case of power loss during conversion (and obviously, having no backups before upgrading violates the First Rule of Sysadmin, but that's the user's fault).

This raises the question: why do we flush the data at all? If power loss destroys data, any flush loses speed for no gain — and flushes make the conversion drastically slower. A simple test case takes more than a day when backed by a SSD — and even on real pmem, there's a large loss of speed. This actually decreases data safety: conversion that takes longer has faces a bigger risk of getting interrupted by power loss.

Thus, pmdk-convert shouldn't flush at all. Not even on real pmem.

Compilation errors on Windows

There are compilation errors on Windows 10 machine.

Building was done according to the README:

• CMake version: 3.17.0
• Windows SDK version: 10.0.18362.1
• Windows version: 1809 (OS Build 17763.1158)
• pmdk-convert revision: d7d9852

Steps to reproduce:

1. Clone pmdk-convert repo:

PS > git clone https://github.com/pmem/pmdk-convert.git

2. Create build directory and select it as the current working directory:

PS > mkdir build
PS > cd build

3. Generate solution:

PS build> cmake .. -G "Visual Studio 14 2015 Win64"

...or point explicitly the Windows SDK version:

PS build> cmake .. -G "Visual Studio 14 2015 Win64" -DCMAKE_SYSTEM_VERSION="10.0.18362.1"

4. Build solution:

PS build> msbuild ALL_BUILD.vcxproj

Expected result:
Solution is compiling successfully without compilation errors.

Actual result:
Solution is not compiling successfully. Produced compilation errors:

(ClCompile target) ->
  C:\Users\test-user\C2\pmdk-convert\1.8-rc2\src\libpmem\pmem_windows.c(65): error C2065: 'WIN32_MEMORY_REGION_INFORMATION': undeclar
ed identifier [C:\Users\test-user\C2\pmdk-convert\build\libpmem-convert.vcxproj]
  C:\Users\test-user\C2\pmdk-convert\1.8-rc2\src\libpmem\pmem_windows.c(65): error C2146: syntax error: missing ';' before identifier
 'region_info' [C:\Users\test-user\C2\pmdk-convert\build\libpmem-convert.vcxproj]
  C:\Users\test-user\C2\pmdk-convert\1.8-rc2\src\libpmem\pmem_windows.c(65): error C2065: 'region_info': undeclared identifier [C:\Us
ers\test-user\C2\pmdk-convert\build\libpmem-convert.vcxproj]
  C:\Users\test-user\C2\pmdk-convert\1.8-rc2\src\libpmem\pmem_windows.c(83): error C2065: 'MemoryRegionInfo': undeclared identifier [
C:\Users\test-user\C2\pmdk-convert\build\libpmem-convert.vcxproj]
  C:\Users\test-user\C2\pmdk-convert\1.8-rc2\src\libpmem\pmem_windows.c(83): error C2065: 'region_info': undeclared identifier [C:\Us
ers\test-user\C2\pmdk-convert\build\libpmem-convert.vcxproj]
  C:\Users\test-user\C2\pmdk-convert\1.8-rc2\src\libpmem\pmem_windows.c(84): error C2065: 'region_info': undeclared identifier [C:\Us
ers\test-user\C2\pmdk-convert\build\libpmem-convert.vcxproj]
  C:\Users\test-user\C2\pmdk-convert\1.8-rc2\src\libpmem\pmem_windows.c(85): error C2065: 'region_info': undeclared identifier [C:\Us
ers\test-user\C2\pmdk-convert\build\libpmem-convert.vcxproj]
  C:\Users\test-user\C2\pmdk-convert\1.8-rc2\src\libpmem\pmem_windows.c(85): error C2224: left of '.DirectMapped' must have struct/un
ion type [C:\Users\test-user\C2\pmdk-convert\build\libpmem-convert.vcxproj]
  C:\Users\test-user\C2\pmdk-convert\1.8-rc2\src\windows\win_mmap.c(546): error C2065: 'FILE_DAX_VOLUME': undeclared identifier [C:\U
sers\test-user\C2\pmdk-convert\build\libpmem-convert.vcxproj]

Detailed log from powershell in attachment:
powershell.log

Release 1.8

When is the 1.8 release planned ?
pmdk 1.8 has been out for a month now and it seems the work has already been done in pmdk-convert for 1.8 support.
Is there something blocking ? Or just a tag missing ?

Missing Coverity scans

In pmdk-convert repo we're missing Coverity scans. It is probably be a good idea to add it, excluding pmdk code.

By this addition it should also be fixed usage of "COVERITY" variable (see similar change for e.g. libpmemobj-cpp repo) - see attachment : coverity_variable_usage.patch.txt

tx_intr_poolset tests are failing on Fedora

ISSUE: tx_intr_poolset tests are failing on Fedora

Environment Information

• PMDK package version(s): e25fc87
• OS(es) version(s): Fedora 34
• kernel version(s): 5.13.13-200
• CMake version: 3.20.5

Please provide a reproduction of the bug:

1. Clone PMDK-convert repo
2. Create build directory and select it as the current working directory
3. Generate solution with CMake options:
   $ cmake .. -DCMAKE_INSTALL_PREFIX=<install_directory> -DDEVELOPER_MODE=1 -DTRACE_TESTS=1 -DCMAKE_BUILD_TYPE=Release -DTESTS_USE_FORCED_PMEM=ON -DDEVICE_DAX_PATHS=\"/dev/dax0.0 /dev/dax0.17"
   4.Run tests:
   ctest -T test --test-output-size-passed 307200 --test-output-size-failed 307200 --no-compress-output --output-on-failure

How often bug is revealed: (always, often, rare): always

Actual behavior:

76% tests passed, 6 tests failed out of 25
Total Test time (real) = 665.40 sec
The following tests FAILED:
20 - tx_intr_pool_devdax (Failed)
21 - tx_intr_poolset_local_devdax (Failed)
22 - tx_intr_poolset_local_2devdaxes (Failed)
23 - tx_intr_poolset_local_devdax_rep_devdax (Failed)
24 - tx_intr_poolset_local_2devdaxes_rep_2regfiles (Failed)
25 - tx_intr_poolset_local_2regfiles_rep_2devdaxes (Failed)
Errors while running CTest

Expected behavior:

Tests should pass

Details

console.log.txt

`pmdk-convert` doesn't support PMDK minor versions >=10 (PMDK v1.10 or later)

Since PMDK v1.10 is now available, pmdk-convert should be updated to support two-digit minor versions to avoid the following error:

# /usr/local/bin/pmdk-convert --from=1.9 --to=1.10 /dev/dax0.0
Invalid "to" version format [major.minor].
Usage: pmdk-convert [--version] [--help] [--no-warning] --from=<version> --to=<version> <pool>

The pool layout created by PMDK v1.10 using pmempool create creates a pool major version = 6. However, pmdk-convert should still support PMDK v1.10 and later.

The error originates from conv_version() that expects X.Y versions, not X.YZ due to the length check on line 293.

static int
conv_version(const char *strver)
{
	if (strlen(strver) != 3)
		return -1;

Failing tests: `pmempool_rm failed: removing file failed`

After merge of #88 there are 6 tests failures:

• tx_intr_pool_devdax,
• tx_intr_poolset_local_devdax,
• tx_intr_poolset_local_2devdaxes,
• tx_intr_poolset_local_devdax_rep_devdax,
• tx_intr_poolset_local_2devdaxes_rep_2regfiles,
• tx_intr_poolset_local_2regfiles_rep_2devdaxes.

Reproduction on pmdk-convert revision: dbe788a.
Failures does not occur on revision: d7d9852.

Configuration "release" and "debug" are affected.

Tested (and affected) OSes:

• Red Hat Enterprise Linux 8.2 with 4.18.0-193.1.2.el8_2.x86_64 kernel,
• Fedora 31 (Server Edition) with 5.7.9-100.fc31.x86_64 kernel.

Namespace configuration:

ndctl list -X
[
  {
    "dev":"namespace0.2",
    "mode":"devdax",
    "map":"dev",
    "size":1054867456,
    "uuid":"344d27ea-e2c5-449f-b5f1-3091994c235c",
    "daxregion":{
      "id":0,
      "size":1054867456,
      "align":4096,
      "devices":[
        {
          "chardev":"dax0.2",
          "size":1054867456,
          "target_node":2,
          "mode":"devdax"
        }
      ]
    },
    "align":4096
  },                                                                                                                                
  {                                                                                                                                 
    "dev":"namespace0.0",                                                                                                           
    "mode":"devdax",                                                                                                                
    "map":"dev",                                                                                                                    
    "size":1054867456,                                                                                                              
    "uuid":"d52f9357-3f0b-42f0-90e2-db323fadceaf",                                                                                  
    "daxregion":{                                                                                                                   
      "id":0,                                                                                                                       
      "size":1054867456,                                                                                                            
      "align":4096,                                                                                                                 
      "devices":[                                                                                                                   
        {                                                                                                                           
          "chardev":"dax0.0",                                                                                                       
          "size":1054867456,                                                                                                        
          "target_node":2,                                                                                                          
          "mode":"devdax"                                                                                                           
        }                                                                                                                           
      ]                                                                                                                             
    },                                                                                                                              
    "align":4096                                                                                                                    
  }                                                                                                                                 
]

Steps to reproduce:

1. Build pmdk-convert:

$ cd /WORKSPACE

$ git clone https://github.com/pmem/pmdk-convert.git

$ cd pmdk-convert

$ git checkout dbe788aec74b44d19941fb7d70321cc77e5c648b

$ mkdir install

$ mkdir build

$ cd build

$ cmake .. -DCMAKE_INSTALL_PREFIX=/WORKSPACE/install -DDEVELOPER_MODE=1 -DTRACE_TESTS=1 -DCMAKE_BUILD_TYPE=RELEASE -DTESTS_USE_FORCED_PMEM=ON -DDEVICE_DAX_PATHS="/dev/dax0.0 /dev/dax0.2"

$ make -j$(nproc)

2. Run pmdk-convert tests:

$ ctest -T test --test-output-size-passed 307200 --test-output-size-failed 307200 --no-compress-output --output-on-failure

Expected result:
The whole tests suite is passing without errors.

Actual result:
Tests (tx_intr_pool_devdax, tx_intr_poolset_local_devdax, tx_intr_poolset_local_2devdaxes, tx_intr_poolset_local_devdax_rep_devdax, tx_intr_poolset_local_2devdaxes_rep_2regfiles, tx_intr_poolset_local_2regfiles_rep_2devdaxes) are failing with the following message:

...
-- Stdout:

/home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/helpers.cmake(71):  file(READ /home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/build-Debug/tests/tx_intr_pool_devdax/err ERR )
/home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/helpers.cmake(72):  message(STATUS Stderr:
pmempool_rm failed: removing file failed
 )
-- Stderr:
pmempool_rm failed: removing file failed

/home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/helpers.cmake(74):  if(NOT RET EQUAL expectation )
/home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/helpers.cmake(75):  message(FATAL_ERROR /home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/build-Debug/tests/clean_pool /dev/dax0.0 exit code 1 doesn't match expectation 0 )
CMake Error at /home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/helpers.cmake:75 (message):
  
  /home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/build-Debug/tests/clean_pool
  /dev/dax0.0 exit code 1 doesn't match expectation 0
Call Stack (most recent call first):
  /home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/helpers.cmake:80 (execute_arg)
  /home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/tx_intr_pool_devdax.cmake:8 (execute)
  /home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/helpers.cmake:229 (prepare_files)
  /home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/tx_intr_pool_devdax.cmake:30 (test_intr_tx_devdax)
  /home/jenkins-slave/workspace/PMDK-convert-linux/pmdk-convert/tests/tx_intr_pool_devdax.cmake:39 (test_devdax)

Log from tests: tests.log

Whole log with all execution details: convert-execution.log

tx_intr_poolset_local_2devdaxes fails

ISSUE:

Environment Information

• OS(es) version(s): Ubuntu 19.10
• ndctl version(s): 65+
• kernel version(s): 5.3.0-26-generic

Please provide a reproduction of the bug:

-- Executing: /pmdk-convert/build/tests/../tests//transaction_14 /pmdk-convert/build/tests/test//🟟⠝⠧⠍⠇ɗPMDKӜ⥺🟟tx_intr_poolset_local_2devdaxes/pool13;vc;7
-- Test /pmdk-convert/build/tests/../tests//transaction_14:
-- Stdout:

-- Stderr:

CMake Error at /pmdk-convert/tests/helpers.cmake:103 (message):
/pmdk-convert/build/tests/../tests//transaction_14
/pmdk-convert/build/tests/test//🟟⠝⠧⠍⠇ɗPMDKӜ⥺🟟tx_intr_poolset_local_2devdaxes/pool13;vc;7
exit code 16 doesn't match expectation 0
Call Stack (most recent call first):
/pmdk-convert/tests/helpers.cmake:108 (execute_arg)
/pmdk-convert/tests/helpers.cmake:263 (execute)
/pmdk-convert/tests/tx_intr_poolset_local_2devdaxes.cmake:64 (test_intr_tx_devdax)
/pmdk-convert/tests/tx_intr_poolset_local_2devdaxes.cmake:73 (test_devdax)

How often bug is revealed: (always, often, rare): always

Actual behavior:

Test fails

Expected behavior:

Test should pass

Details