libyal/libewf

Libewf is a library to access the Expert Witness Compression Format (EWF)

libewf's Issues

Tests don't run if the build directory contains the version number.

For building the Debian package using sbuild, I patched tests/test_*.sh so that they could be successfully run in a build directory .../libewf-20160325:

-TEST_PREFIX=`basename ${TEST_PREFIX} | sed 's/^lib//'`;
+TEST_PREFIX=`basename ${TEST_PREFIX} | sed 's/^lib//' | sed 's/-201.*$//'`;

(This change was also necessary for the recent libsigscan release.)
Wouldn't it be better to just hard-code TEXT_PREFIX?

libewf 20150126 problems handling E01 images with > 64 file segments

I have multiple examples of E01 images, created using FTK Imager, with more than 64 file segments. The current version of ewfverify fails on these images. However, ewfverify 20140608 works correctly when opening and verifying the images.

Steps to reproduce:

1. create two E01 images of the same physical drive, such that the first image has 64 segments or less (i.e., the last segment having an extension of .E64 or lower) and the second image has 65 segments or more (ending with .E65 or higher).
2. Use a separate tool to verify the hash values of the data in each image.
3. Use ewfverify 20140608 to verify each image. Both should be successful.
4. Use ewfverify 20150126 to verify each image. The first should succeed and the second should fail.

I note that, in December 2014, the constant LIBEWF_MAXIMUM_CACHE_ENTRIES_SEGMENT_FILES was changed to 64.
https://github.com/libyal/libewf/blame/master/libewf/libewf_definitions.h.in#L440
Is it possible that this change to the size of the segment_files_cache in a segment_table could be related?

fix issues with writing/reading

• fix #8 - added test coverage
• fix #22 - handling split images
  • determine why chunks are seen as corrupt, also for EWF version 1
• update man pages #27
• fix #29
  • fixed unmanaged "corrupted" chunk data
• fix last chunk write issue in split images
• determine why read falls back to table2
• improve handling missing chunks list
• restrict resources used by multi-threaded approach
• test and fix issues with new read/write chunk API
• look into report of ewfverify -l not logging anything?
• look into #35
• look into #93
• look into slow write chunk for version 2 format
• complete chunk group size limitation

Fails to build with recent libcstring

Build on Fedora linux fails with the undefined reference to the libcstring_narrow_string_length and libcstring_narrow_string_copy. It seems that such function is no longer in libcstring.

I have tried with both release 20150126 (5e47a75) and current head commit (54b0ead)

This is the error message I get during build.

/bin/sh ../libtool  --tag=CC   --mode=link gcc  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Wall  -Wl,-z,relro  -o ewfacquirestream byte_size_string.o digest_hash.o ewfacquirestream.o ewfinput.o ewfoutput.o guid.o imaging_handle.o log_handle.o platform.o process_status.o storage_media_buffer.o -luuid -lhmac -lcsystem -lcsplit -lcdatetime ../libewf/libewf.la -lcnotify -lclocale -lcerror -lcstring  -lbz2 
/bin/sh ../libtool  --tag=CC   --mode=link gcc  -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Wall  -Wl,-z,relro  -o ewfdebug byte_size_string.o ewfdebug.o ewfinput.o ewfoutput.o -lcsystem ../libewf/libewf.la -lcnotify -lclocale -lcerror -lcstring  -lbz2 
libtool: link: gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wall -Wl,-z -Wl,relro -o .libs/ewfacquire byte_size_string.o digest_hash.o device_handle.o ewfacquire.o ewfinput.o ewfoutput.o guid.o imaging_handle.o log_handle.o platform.o process_status.o storage_media_buffer.o  -lodraw -lsmdev -lsmraw -luuid -lcsystem -lcdatetime ../libewf/.libs/libewf.so -L/usr/lib64 -lcdata -lcsplit -luna -lcfile -lcpath -lbfio -lfcache -lfdata -lfvalue -lz -lhmac -lcaes -lcnotify -lclocale -lcthreads -lcerror -lcstring -lbz2
libtool: link: gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wall -Wl,-z -Wl,relro -o .libs/ewfacquirestream byte_size_string.o digest_hash.o ewfacquirestream.o ewfinput.o ewfoutput.o guid.o imaging_handle.o log_handle.o platform.o process_status.o storage_media_buffer.o  -luuid -lcsystem -lcdatetime ../libewf/.libs/libewf.so -L/usr/lib64 -lcthreads -lcdata -lcsplit -luna -lcfile -lcpath -lbfio -lfcache -lfdata -lfvalue -lz -lhmac -lcaes -lcnotify -lclocale -lcerror -lcstring -lbz2
libtool: link: gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wall -Wl,-z -Wl,relro -o .libs/ewfdebug byte_size_string.o ewfdebug.o ewfinput.o ewfoutput.o  -lcsystem ../libewf/.libs/libewf.so -L/usr/lib64 -lcthreads -lcdata -lcsplit -luna -lcfile -lcpath -lbfio -lfcache -lfdata -lfvalue -lz -lhmac -lcaes -lcnotify -lclocale -lcerror -lcstring -lbz2
../libewf/.libs/libewf.so: undefined reference to `libcstring_narrow_string_length'
../libewf/.libs/libewf.so: undefined reference to `libcstring_narrow_string_copy'
collect2: error: ld returned 1 exit status
Makefile:1091: recipe for target 'ewfacquire' failed
make[1]: *** [ewfacquire] Error 1
make[1]: *** Waiting for unfinished jobs....
ewfacquirestream.o: In function `ewfacquirestream_read_chunk':
/home/mambroz/rpmbuild/BUILD/libewf-54b0eada69defd015c49e4e1e1e4e26a27409ba3/ewftools/ewfacquirestream.c:326: undefined reference to `libcsystem_file_io_read'
../libewf/.libs/libewf.so: undefined reference to `libcstring_narrow_string_length'
../libewf/.libs/libewf.so: undefined reference to `libcstring_narrow_string_copy'
collect2: error: ld returned 1 exit status
Makefile:1095: recipe for target 'ewfacquirestream' failed
make[1]: *** [ewfacquirestream] Error 1
../libewf/.libs/libewf.so: undefined reference to `libcstring_narrow_string_length'
../libewf/.libs/libewf.so: undefined reference to `libcstring_narrow_string_copy'
collect2: error: ld returned 1 exit status
Makefile:1099: recipe for target 'ewfdebug' failed
make[1]: *** [ewfdebug] Error 1
make[1]: Leaving directory '/home/mambroz/rpmbuild/BUILD/libewf-54b0eada69defd015c49e4e1e1e4e26a27409ba3/ewftools'
Makefile:804: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.nGlIJ0 (%build)

ewfverify fails on large drives on 32 bit systems

Ewfverify fails on large images. I've used ewfverify on the images of a large (3TB) disk, and ewfverify consistently fails at the same point. The error messages suggest there is a problem with reading the evidence files, but the images are verified correctly in EnCase.

I've tried outputting more verbose error logs, but the resulting log file stays empty. I've used the latest version (libewf-20140608.tar.gz), and built it in Ubuntu 12.04 x86, and the latest Mint (17.1) x86.

I've used the same version of libewf to verify the evidence files on a 64 bit system, and the verification passes.

Is it perhaps a limitation of 32 bit systems, or is there a possible bug?

The error messages from the console:

Status: at 38%.
        verified 1.0 TiB (1163113988096 bytes) of total 2.7 TiB (3000592982016 bytes).
        completion in 2 hour(s), 56 minute(s) and 32 second(s) with 167 MiB/s (175637613 bytes/second).

Status: at 38%.
        verified 1.0 TiB (1163994595328 bytes) of total 2.7 TiB (3000592982016 bytes).
        completion in 2 hour(s), 56 minute(s) and 38 second(s) with 167 MiB/s (175534864 bytes/second).

Status: at 38%.
        verified 1.0 TiB (1164875137024 bytes) of total 2.7 TiB (3000592982016 bytes).
        completion in 2 hour(s), 56 minute(s) and 45 second(s) with 167 MiB/s (175421980 bytes/second).

Verify failed at: Mar 13, 2015 13:25:09
Unable to verify input.
libewf_handle_read_buffer: unable to read chunk data: 35570143.
verification_handle_read_buffer: unable to read storage media buffer.
verification_handle_verify_input: unable to read data.
ewfverify: FAILURE

Unable to get pyewf, pyqcow, pyvmdk working in plaso

I am running an Arch 64 build:

Linux ARCHIR 4.2.5-1-ARCH #1 SMP PREEMPT Tue Oct 27 08:13:28 CET 2015 x86_64 GNU/Linux

[ Problem ]:
No matter what I do, following your instructions on building from source does not allow me to 'apparently' install the libraries correctly. I say apparently because the PLASO RUNTESTS.PY script reports these libraries as missing.

What should I do?

See below.

$ python2 ./run_tests.py
Checking availability and versions of plaso dependencies.
[OK]            artifacts version: 20150409
[OK]            bencode
[OK]            binplist version: 0.1.4
[OK]            construct version: 2.5.2
[OK]            dateutil version: 2.4.2
[OK]            dfvfs version: 20151231
[OK]            dpkt version: 1.8
[OK]            google.protobuf
[OK]            hachoir_core version: 1.3.3
[OK]            hachoir_parser version: 1.3.4
[OK]            hachoir_metadata version: 1.3.3
[OK]            IPython version: 4.0.0
[OK]            pefile version: 1.2.10-139
[OK]            psutil version: 3.3.0
[OK]            pyparsing version: 2.0.7
[OK]            pytz
[OK]            requests version: 2.9.1
[OK]            six version: 1.10.0
[OK]            sqlite3 version: 3.9.2
[OK]            yaml version: 3.11
[OK]            SleuthKit version: 4.2.0
[OK]            pytsk3 version: 20150406
[OK]            libbde (pybde) version: 20160108
[OK]            libesedb (pyesedb) version: 20160107
[OK]            libevt (pyevt) version: 20160107
[OK]            libevtx (pyevtx) version: 20160107
[FAILURE]       missing: pyewf.
[OK]            libfwsi (pyfwsi) version: 20160106
[OK]            liblnk (pylnk) version: 20160107
[OK]            libmsiecf (pymsiecf) version: 20160107
[OK]            libolecf (pyolecf) version: 20160107
[FAILURE]       missing: pyqcow.
[OK]            libregf (pyregf) version: 20160107
[OK]            libsigscan (pysigscan) version: 20160108
[OK]            libsmdev (pysmdev) version: 20160108
[OK]            libsmraw (pysmraw) version: 20160108
[OK]            libvhdi (pyvhdi) version: 20160108
[FAILURE]       missing: pyvmdk.
[OK]            libvshadow (pyvshadow) version: 20160108

Feature request: Progress display to only update at percentage points.

Would like to have a way to disable the constant progress display entries so that the only progress display entries are for each percentage points. (e.g. for 1% 2% ... etc)

It is really just asking to for a method to get the behaviour of libewf_20100226 or similar as seen in the ewf-tools package in debian squeeze.
( https://packages.debian.org/squeeze/ewf-tools )

This will help me to work around issue 16 where there is an problem with progress updates under windows.
ref: #16

Revisit tests

• revisit test plan
• valgrind tests
• fix test failing on Cygwin
  • glob test not finishing (horribly slow on Cygwin - skip for now on Cygwin, until faster version)
• fix test failing on MinGW-MSYS
  • glob test failing
  • fix sed: -e expression #1, char 9: unterminated `s' command
• afl tests - now covered by OSSFuzz
• check / fix prefix in test scripts
• Add bounds tests
  • number of segment files
  • max supported input size
• single threaded and multi threaded tools tests
• re-add interactive mode for tools tests
• pyewf write test
• improve performance of glob test to run on Cygwin
• speed comparison tests
  • /dev/urandom using dd and ewfacquire
  • /dev/zero using dd and ewfacquire

libewf-experimental-<version>.tar.gz file not on downloads page.

Unable to find the "libewf-experimental-.tar.gz" on the downloads page following the building instructions in below link.

https://github.com/libyal/libewf/wiki/Building
"To retrieve the source package go to the downloads page and download the file named:
libewf-experimental-.tar.gz"

The downloads page is in the following link:
https://googledrive.com/host/0B3fBvzttpiiSMTdoaVExWWNsRjg/

replace delta segment file support by more generic solution

The delta segment file solution is only useful for a small number of changes and adds a significant amount of complexity to the code base.

• remove delta segment file support
• create a generic solution (library + tooling) to create a writable overlay outside libewf

wrong compression flag and chunk padding cause verification failure

Data chunk is not properly aligned when running ewfacquire with '-x' switch. The resulting Ex01 file layout is different from which was created without the '-x' switch, and cannot be verified successfully by ewfverify tool.

ewfacquire -x -C test -u -q -d sha1 -m fixed -f encase7-v2 -c fast -t /tmp/test /dev/sdc
"ewfverify /tmp/test" failed with message "unexpected end of data".

A quick tracing of the code shows two problems:

1. using LIBEWF_CHUNK_DATA_FLAG_IS_COMPRESSED with chunk_data->range_flags seems to be wrong, which should be LIBEWF_RANGE_FLAG_IS_COMPRESSED instead.
2. In libewf_handle_prepare_write_chunk(), variable chunk_padding_size never get used. It should be passed somehow to libewf_handle_write_chunk().

What pre-dated libewf_set_volume_type, if anything?

I note the function : libewf_set_volume_type (

libewf/libewf/libewf_legacy.c

Line 2373 in 54b0ead

int libewf_set_volume_type(

) and the accompanying byte values of

LIBEWF_VOLUME_TYPE_LOGICAL = 0x00,
LIBEWF_VOLUME_TYPE_PHYSICAL = 0x01
(

libewf/libewf/libewf_definitions.h.in

Line 162 in 54b0ead

enum LIBEWF_VOLUME_TYPES

)

From what I can gather, these should be passed to libewf_set_volume_type so that the resulting E01 'knows' whether it is an E01 of a physical item (.\PhysicalDiskX) or logical volume?

However, the compiled DLL I have dates back to June 2014 and using DLL Explorer, the function libewf_set_volume_type is not listed in it, so I assume it dates later than my DLL.

Was there another function that predated libewf_set_volume_type but which did the same thing? If so, could you tell me what it was called? I can't find anything that reselmbles it - libewf_set_media_type is the nearest but I think that's things like "CD\USB\Disk" etc I realise the best thing would be to generate a new DLL from the latest source code but having tried that the other day, I found it didn't work with my project (I use Delphi\Freepascal).

Thanks

ewfacquire "resume" reimages source file

When trying to reproduce steps from test_ewfacquire_resume.sh I noticed that source file is re-acquired from the beggining (the "status" messages start from 0%) when ewfacquire is called with -R flag. Also ewfacquire determines "Resume acquiry offset" from "underimaged" file as 0, which also seems weird to me, however confirms strange behaviour. This behaviour seems more like "restart" but not "resume".

Also when using .NET bindings, library behaves the same way:

1. I open handle, set media size to, say, 1GB.
2. Write half of the size, close the handle.
3. Reopen it with "WriteResume" access flags.
4. Try to write the remaining half (seeking to the 500MB), but witness that it writes form the beginning of the file, overwriting previously written data, as if no seeking was performed.

• Affected by versions: 20150126 (from the repo), 20140608 (from downloads)
• Build from MSVS 2013. However "./configure", "make", "make dist" were made on CentOS, then I moved files to Windows machine.
  OS: Windows 8 64 bit.

Ewfexport not setting format correctly

ewfexport -f encase6 exports to RAW which is not the correct behavior

ewfacquire: memory exhaustion

A quick test run of ewfacquire caused all system memory to be quickly used up and the process was terminated:

ewfacquire -C test -u -q -d sha1 -m fixed -f encase7-v2 -c fast -t /tmp/test /dev/sdc

It turns out to be that chunk_data allocated in libewf_handle.c:libewf_handle_write_buffer() never gets freed.

ewfmount not releaseing with umount

I mounted a small multi-partition .E01 image to /media/mntpt,
used
kpartx -a to create the /dev/mapper/loop0p1.
then
mount /dev/mapper/loop0p1 -o ro /media/mntpt1

viewed two graphics files

umount /dev/mapper/loop0p1
kpartx -d /dev/mapper/loop0p1

then tried to umount /media/mntpt
and got the error:
"umount: /media/mntpt: target is busy"

ewfmount 20150107 failing to mount evidence in readable format

I'm attempting to mount an image of a 500GB hard drive acquired using EnCase 6.19.7. After running the command to mount (ewfmount /mnt/hgfs/Cases/evidence.e* /mnt/ewf/), all seems to be good and I get no error messages. ewf1 appears in the specified location. When I try parted or mmls, it is unable to recognise any partitions.

I have successfully mounted the same image using ewfmount 20140608.

Happy to share any info you need.

incomplete format list in ewfacquire and ewfexport man pages

ewfacquire -h and ewfexport -h list encase7-v2 among the possible arguments for the -f flag. The man pages for ewfacquire and ewfexport do not list encase7-v2 as one of the arguments for the -f flag. The man page should be updated to reflect what's supported.

Building libewf-experimental-20160403 using MSYS-MinGW on Windows 7 64-bit

Hi Joachim

I wanted to try using a new DLL compiled from your recent March release. It has been a while since I compiled it last time so it took me a while some reading to do, but having done it, I fear I have done something wrong.
I followed the instructions here to build using Windows 7 64-bit and MSYS-MinGW. Upon completion I had libewf-3.dll and I also went and downloaded libgcc_s_dw2-1.dll from somewhere and copied that to my project folder, along with my existing zlib.dll file and so on.

Trouble is, my LoadLibraryA call fails. I thought for a while it was due to something I had done, but after a few hours debugging and using Dependancy Walker, I realised that the newly compiled DLL was missing dependancies, namely libbz2-2.dll and zlib1.dll (I have a zlib.dll already, just not zlib1.dll) to start with. But even after downloading those, there remained several other warnings in Dependancy Walker stating that "Error opening file. The system cannot find the file specified" for the following files:

API-MS-WIN-APPMODEL-RUNTIME-L1-1-0.DLL
API-MS-WIN-CORE-WINRT-ERROR-L1-1-0.DLL
API-MS-WIN-CORE-WINRT-L1-1-0.DLL
API-MS-WIN-CORE-WINRT-ROBUFFER-L1-1-0.DLL
API-MS-WIN-CORE-WINRT-STRING-L1-1-0.DLL
API-MS-WIN-SHCORE-SCALING-L1-1-1.DLL
DCOMP.DLL
IESHIMS.DLL

So at this stage, I gave up and thought I'd ask you here in case I was missing something obvious.
Have you compiled the latest libEWF yourself using Windows and MSYS-MinGW? If so, can I ask what else you did to ensure your other projects or any programs can load the resulting libewf.DLL? As it stands, I have the following DLL's in my project folder :

libbz2-2.dll (downloaded from Internet)
libewf.dll (that I renamed from libewf-3.dll, built with MSYS-MinGW)
libgcc_s_dw2-1.dll (downloaded from Internet)
msvcr100.dll (downloaded from Internet)
zlib.dll (about a year or two old)
zlib1.dll (downloaded from Internet, dated 2013)

Many thanks as always for your time and development work.

Time estimate appears to use integer GB to calculate estimate

Using ewfacquire 20140608 (not the newer experimental release).

I happen to notice the initial time estimate can be way off, but then it jumps drastically on the 1GB updates.

It's like the code does something like:

time_remaining = total_GB * (time_in_progress / int(GB_done))

Thus in the first few GB the error is pretty drastic. In particular, my 2 1/2 hour image was estimating 5 hours at the 1.9 GB process stage, but then corrected to 2 1/2 hours on the next update.

Struggling assigning NONE, FAST or BEST with libewf_set_compression_values

Hi Joachim

I'm having some difficulty applying compression. I am using libewf_set_compression_values to assign BEST, LOW, NONE compression to E01 image files (though I notice there's a 'libewf_handle_set_compression_values - perhaps I need to use that?)

libewf/libewf/libewf_metadata.c

Line 732 in 9f67838

int libewf_handle_set_compression_values(

I am also assigning LIBEWF_COMPRESSION_LEVELS within my function. (https://github.com/libyal/libewf/blob/9f678389d18b46db96c6a3b169a46afc2516bf39/include/libewf/definitions.h.in)

The user is asked to choose a default, none, fast or best compression method via the GUI and their selection is assigned to an integer var as -1, 0, 1 or 2.

I then lookup that variables value and,

if it's -1, libewf_set_compression_values is called with (LIBEWF_COMPRESSION_DEFAULT, 0).

if it's 0, libewf_set_compression_values is called with (LIBEWF_COMPRESSION_NONE, 0).

if it's 1, libewf_set_compression_values is called with (LIBEWF_COMPRESSION_FAST, 0).

if it's 2, libewf_set_compression_values is called with (LIBEWF_COMPRESSION_BEST, 0).

If the user chooses none, I get an uncompresssed E01 just as expected and hoped. However, if the user chooses FAST or BEST, the resulting image is always the same size. I would expect one to be larger than the other, even if by only a small amount.

Do I have to call something else? In other words, having SET the compression with libewf_set_compression_values, do I need to call another function to actually incorporate the varying levels of compression?

Status information updating too often under Windows

environment

Source code version : libewf-20140608.tar.gz
(have also tried with latest from git (cloned yesterday))
OS: windows 7 64 bit

built under cygwin using following configure command (as well as usual make and make install)
./configure --host=i686-w64-mingw32 --enable-static-executables --enable-python

ewfacquire .\PhysicalDrive1
Worked fine but with similar issues with status updating too soon.

These tools work perfectly fine apart from the status update which spams the screen with updates as well as the transfer rate being incorrect.

misc note:

1. the status update oddly affects performance more under cmd.exe as compared to when being run under cygwin terminal
2. didn't have python dev files install so enable python was ignored. (as per .configure output)

Issue

Status information is updating too often. (example below)
Affects the performance as well as the transfer rate seems affected.

< begin snip>
Status: at 20%.
verified 829 MiB (869826560 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

Status: at 20%.
verified 829 MiB (869924864 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

Status: at 20%.
verified 829 MiB (870055936 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

Status: at 20%.
verified 829 MiB (870154240 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

Status: at 20%.
verified 829 MiB (870252544 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

Status: at 20%.
verified 829 MiB (870285312 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

Status: at 20%.
verified 830 MiB (870416384 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

Status: at 20%.
verified 830 MiB (870547456 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

Status: at 20%.
verified 830 MiB (870711296 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

Status: at 20%.
verified 830 MiB (870875136 bytes) of total 3.9 GiB (4227858432 bytes).
completion in 33 second(s) with 10 bytes/second.

dpkg: fix default config not conforming to Ubuntu and Debian package naming

Check if a Replaces section can fix dependency problems on Ubuntu with libewf2
https://www.debian.org/doc/debian-policy/ch-relationships.html#s-replaces

libewf_handle_set_sha1_hash not storing the hash in the E01

Hi

I am using libewf_handle_set_md5_hash and libewf_handle_set_sha1_hash to embed either Md5 or SHA1 hash values into the E01 using libewf.

libewf_handle_set_md5_hash works fine - the MD5 hash is embedded. But when I use the same technique with libewf_handle_set_sha1_hash, the function returns valid (no error, integer returned is 1) but when I open the E01 file using FTK Imager, the hash value properties field is empty.

What might I be doing wrong? Or is there a problem with the function? I am using the libewf.dll from http://labalec.fr/erwan/?p=1235 (specifically this download http://labalec.fr/erwan/wp-content/uploads/2014/05/libewf2.zip), which may be the problem?

Building Python bindings on Windows 7 throws unresolved symbols from libhmac

Running python setup.py build on Windows 7 after following all other build instructions causes the C++ linker to throw the following errors:

libhmac_md5.obj : error LNK2019: unresolved external symbol __imp__CryptReleaseContext@8 referenced in function _libhmac_md5_initialize
libhmac_sha1.obj : error LNK2001: unresolved external symbol __imp__CryptReleaseContext@8
libhmac_md5.obj : error LNK2019: unresolved external symbol __imp__CryptCreateHash@20 referenced in function _libhmac_md5_initialize
libhmac_sha1.obj : error LNK2001: unresolved external symbol __imp__CryptCreateHash@20
libhmac_md5.obj : error LNK2019: unresolved external symbol __imp__CryptAcquireContextW@20 referenced in function _libhmac_md5_initialize
libhmac_sha1.obj : error LNK2001: unresolved external symbol __imp__CryptAcquireContextW@20
libhmac_md5.obj : error LNK2019: unresolved external symbol __imp__CryptDestroyHash@4 referenced in function _libhmac_md5_free
libhmac_sha1.obj : error LNK2001: unresolved external symbol __imp__CryptDestroyHash@4
libhmac_md5.obj : error LNK2019: unresolved external symbol __imp__CryptHashData@16 referenced in function _libhmac_md5_update
libhmac_sha1.obj : error LNK2001: unresolved external symbol __imp__CryptHashData@16
libhmac_md5.obj : error LNK2019: unresolved external symbol __imp__CryptGetHashParam@20 referenced in function _libhmac_md5_finalize
libhmac_sha1.obj : error LNK2001: unresolved external symbol __imp__CryptGetHashParam@20

This is most likely because of problems in including wincrypt.h, as the functions are declared in the header but apparently not found by anything else. This was attempted with Python 2 and Visual Studio 9.0.

add EWF version 2 bzip2 support

• create test image
• add read support
• add write support

Issue with +2GB LVF

+2GB LVF seems to define 8-byte size rather than a 4-byte
try to reproduce and update documentation

gcc errors

I'm receiving errors when attempting to build from source on an Ubuntu 12.04 machine.

Following the steps:
./synclibs.sh
./autogen.sh
./configure
make

gcc: error: @LIBFDATETIME_CPPFLAGS@: No such file or directory
gcc: error: @LIBFGUID_CPPFLAGS@: No such file or directory
gcc: error: @LIBFWNT_CPPFLAGS@: No such file or directory

The synclibs.sh file makes no reference to these libraries. But I see the Makefile under libfvalue references these on line 540-542 under AM_CPPFLAGS.

Possibly Pertinent Info: I need to build this on an air-gapped machine, so I downloaded the repo on another, Internet-connected machine (running same OS version), ran ./synclibs.sh, and then copied that directory to the target machine and tried to build.

Exporting an image to stdout with ewfexport appends exit status to image body

When using ewfexport to send an image to stdout, it will append the string "ewfexport: SUCCESS" to end of the image on completion. Receiving programs will interpret this string as part of the image body. For example, here someprogram will not be working with an exact copy of a raw image, but has 19 additional bytes at the end...

ewfexport -u -f raw -t - image.E01 | someprogram

Maybe that exit message should go to stderr instead, when stdout is used for the image?

wide character support

Hello, I am sorry if it's not the correct place to ask questions but I couldn't find a place for discussion. I compiled libclocale using these options:

./configure --enable-wide-character-type --enable-shared=no

and libewf using these options:

./configure --enable-verbose-output --enable-shared=no --enable-static-executables=yes --enable-python --enable-wide-character-type

Features:
   Multi-threading support:                  pthread
   Wide character type support:              yes
   ewftools are build as static executables: yes
   Python (pyewf) support:                   2.7
   Verbose output:                           yes
   Debug output:                             no
   Version 1 API compatibility:              no

Then I ewfmount a .E01 file, then mount -o ro,loop to mount it in Ubuntu 14.04. I am seeing question marks like these in ls

????_2014 - 2015?????????????.docx 

where there should be CJK characters. (Ubuntu itself can view CJK characters correctly)

So I guess my question is, are wide characters and Unicode supported? If not, how can I correctly view those files?

flex.exe not responding when running configure

Hi, i need help on building libewf latest release using MSYS-MinGW on Windows 10. When i run CPPFLAGS=-DWINVER=0x0501 ./configure --prefix=/mingw it keep stopping at flex checking and said flex.exe not responding. Do you have any idea on what should i do? Thanks.

libewf: complete EWF2-Ex01 write support

• add EWF2-Ex01 write support

libewf: complete EWF2-Ex01 and EWF2-Lx01 read support

• complete EWF2-Ex01 and EWF2-Lx01 read support

ewfacquire error failure

I am trying to acquiring 8GB USB, and i always got error failure between 90%.
here error output

Acquiry failed at: May 25, 2016 20:17:40
Unable to acquire input.
libcfile_file_read_buffer_with_error_code: invalid read count: 28672 returned.
libsmdev_handle_read_buffer: unable to read from device file.
device_handle_read_storage_media_buffer: unable to read buffer from device input handle.
ewfacquire_read_input: error reading data from input.
ewfacquire: FAILURE

what cause this error?

TP_POOL support not implemented yet on Visual Studio 2013

When I try to build libewf_dll project, I got following error from Visual Studio
TP_POOL support not implemented yet on Visual Studio 2013

The error is caused by following line from libcthreads_thread_pool.c file

if defined( WINAPI ) && ( WINVER >= 0x0602 )

/* TODO */

error TP_POOL support not implemented yet

libewf memory spikes rapidly until system memory is exhausted

I tried to extract a E01 file (http://digitalcorpora.org/corpora/scenarios/2012-ngdc/carry-tablet/carry-tablet-2012-07-16-final.E01) to raw format. The memory used by ewfexport stays at around 40MB initially. It jumped more than 4GB at around 89% and depleted system memory. The following is the output and the version of ewfexport.

$ ewfexport carry-tablet-2012-07-16-final.E01 > /dev/null
ewfexport 20150126

Information for export required, please provide the necessary input
Export to format (raw, files, ewf, smart, ftk, encase1, encase2, encase3, encase4, encase5, encase6, encase7, encase7-v2, linen5, linen6, linen7, ewfx) [raw]: 
Target path and filename without extension or - for stdout: -
Start export at offset (0 <= value <= 30928797696) [0]: 
Number of bytes to export (0 <= value <= 30928797696) [30928797696]: 

Export failed at: Nov 25, 2015 07:21:29
Unable to export input.
libewf_chunk_data_initialize_clear_data: unable to create data.
libewf_chunk_table_get_chunk_data_by_offset: unable to create chunk: 879016 data.
libewf_handle_read_buffer: unable to read chunk: 879016 data.
export_handle_read_buffer: unable to read storage media buffer.
export_handle_export_input: unable to read data.

ewftools UX enhancements

• ewfverify add the name of the image in the log_file and possibly the date and time

ewfacquire fails to image one specific thumb drive

I have successfully imaged other thumb drives so this is not a generic issue.

With a specific thumb drive I am trying to image ewfacquire is failing. The thumbdrive itself is operational and I can image it via dc3dd.

ewfacquire /dev/sdb
ewfacquire 20140608

Device information:
Unable to print media information.
libsmdev_string_trim_copy_from_byte_stream: string too small.
libsmdev_internal_handle_determine_media_information: unable to set serial number.
libsmdev_handle_get_bus_type: unable to determine media information.
device_handle_media_information_fprint: unable to retrieve bus type.
Unable to retrieve media type from device.
libsmdev_string_trim_copy_from_byte_stream: string too small.
libsmdev_internal_handle_determine_media_information: unable to set serial number.
libsmdev_handle_get_media_type: unable to determine media information.
device_handle_get_media_type: unable to retrieve media type.

The thumb drive is a generic one.

l don't know if it is helpful, but "lsusb -v -s" gives this detail

Bus 004 Device 003: ID 13fe:5500 Kingston Technology Company Inc.
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.00
bDeviceClass 0 (Defined at Interface level)
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 9
idVendor 0x13fe Kingston Technology Company Inc.
idProduct 0x5500
bcdDevice 1.00
iManufacturer 1
iProduct 2 Patriot Memory
iSerial 3
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 44
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0x80
(Bus Powered)
MaxPower 126mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 8 Mass Storage
bInterfaceSubClass 6 SCSI
bInterfaceProtocol 80 Bulk-Only
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 3
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x02 EP 2 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 3
Binary Object Store Descriptor:
bLength 5
bDescriptorType 15
wTotalLength 22
bNumDeviceCaps 2
USB 2.0 Extension Device Capability:
bLength 7
bDescriptorType 16
bDevCapabilityType 2
bmAttributes 0x00000002
Link Power Management (LPM) Supported
SuperSpeed USB Device Capability:
bLength 10
bDescriptorType 16
bDevCapabilityType 3
bmAttributes 0x00
wSpeedsSupported 0x000e
Device can operate at Full Speed (12Mbps)
Device can operate at High Speed (480Mbps)
Device can operate at SuperSpeed (5Gbps)
bFunctionalitySupport 2
Lowest fully-functional device speed is High Speed (480Mbps)
bU1DevExitLat 10 micro seconds
bU2DevExitLat 2047 micro seconds
Device Status: 0x0000
(Bus Powered)

windows msys

hi, i got error when building libewf (experimental 20160424) with msys in mingw. i have zlib, bzip2 and Crypto API librra. i follow this article #https://github.com/libyal/libewf/wiki/Building#using-msys-mingw, but fail while make.

Making all in include
make[1]: Entering directory /home/HUDA/libewf-20160424/include' make[1]: Nothing to be done forall'.
make[1]: Leaving directory /home/HUDA/libewf-20160424/include' Making all in common make[1]: Entering directory/home/HUDA/libewf-20160424/common'
make all-am
make[2]: Entering directory /home/HUDA/libewf-20160424/common' make[2]: Leaving directory/home/HUDA/libewf-20160424/common'
make[1]: Leaving directory /home/HUDA/libewf-20160424/common' Making all in libcstring make[1]: Entering directory/home/HUDA/libewf-20160424/libcstring'
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -Wall -o libcstring.la libcstring_support.lo libcstring_wide_string.lo -lbz2 -lz
libtool: link: rm -fr .libs/libcstring.lax
libtool: link: lib -OUT:.libs/libcstring.lib .libs/libcstring_support.o .libs/libcstring_wide_string.o
make[1]: Leaving directory `/home/HUDA/libewf-20160424/libcstring'

and error messages
../libtool: line 1095: lib: command not found
make[1]: *** [libcstring.la] Error 127
make: *** [all-recursive] Error 1

i google

lib: command not found

but no one work. any one know this issue?
my OS is win7 64-bit, and i am not sure if i install mingw 64 bit (there is mingw32 folder inside mingw directory). i don't have any problem while build libewf (same version) in centos. please helpme.

Add fallback deflate and inflate implementation

• Add fallback inflate implementation when for when zlib is not available
• Add fallback deflate implementation when for when zlib is not available

build fails under cygwin

Just tried to download and build current rev under cygwin. It fails with the following error:

make[2]: Entering directory '/cygdrive/c/Users/jm460/Downloads/libewf/libewf-master/libodraw'
/bin/sh ../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I../common  -I../include -I../common -I../libcstring -I../libcerror -I../libcthreads -I../libcdata -I../libclocale -I../libcnotify -I../libcsplit -I../libuna -I../libcfile -I../libcpath -I../libbfio  -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -D_FILE_OFFSET_BITS=64  -g -O2 -Wall -MT libodraw_cue_parser.lo -MD -MP -MF .deps/libodraw_cue_parser.Tpo -c -o libodraw_cue_parser.lo libodraw_cue_parser.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I../common -I../include -I../common -I../libcstring -I../libcerror -I../libcthreads -I../libcdata -I../libclocale -I../libcnotify -I../libcsplit -I../libuna -I../libcfile -I../libcpath -I../libbfio -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -MT libodraw_cue_parser.lo -MD -MP -MF .deps/libodraw_cue_parser.Tpo -c libodraw_cue_parser.c  -DDLL_EXPORT -DPIC -o .libs/libodraw_cue_parser.o
libodraw_cue_parser.c: In function 'cue_scanner_parse':
libodraw_cue_parser.c:1544:7: error: too few arguments to function 'cue_scanner_lex'
       yychar = yylex ();
       ^
libodraw_cue_parser.c:416:12: note: declared here
 extern int cue_scanner_lex(
            ^
libodraw_cue_parser.c:1658:30: error: 'parser_state' undeclared (first use in this function)
     ( (cue_parser_state_t *) parser_state )->error,
                              ^
libodraw_cue_parser.c:1658:30: note: each undeclared identifier is reported only once for each function it appears in
libodraw_cue_parser.c: In function 'cue_parser_parse_buffer':
libodraw_cue_parser.c:2742:8: error: too many arguments to function 'cue_scanner_parse'
        &parser_state ) == 0 )
        ^
libodraw_cue_parser.c:63:25: note: declared here
 #define yyparse         cue_scanner_parse
                         ^
libodraw_cue_parser.c:1388:1: note: in expansion of macro 'yyparse'
 yyparse (void)
 ^
Makefile:784: recipe for target 'libodraw_cue_parser.lo' failed
make[2]: *** [libodraw_cue_parser.lo] Error 1
make[2]: Leaving directory '/cygdrive/c/Users/jm460/Downloads/libewf/libewf-master/libodraw'
Makefile:697: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/cygdrive/c/Users/jm460/Downloads/libewf/libewf-master/libodraw'
Makefile:824: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

Also, it would be really nice if there was an approved repository of precompiled downloads for Windows. Not everybody has windows compilers other than gnu, and the cygwin build has never worked as far as I'm aware.
Thanks
John

ewfverify: memory exhaustion

Create encase-7 ex01 files for a 512GB dm-zero device:

echo '0 1000000000 zero' | dmsetup create z1
ewfacquire -C test -u -q -d sha1 -m fixed -f encase7-v2 -c fast -S 200000000 -t test /dev/mapper/z1

This created 7 files, test.Ex01...Ex07, each has a size of nearly 500MB.
Run ewfverify against these files:

ewfverify test.Ex01

The memory usage of ewfverify process grows linearly over time and become nearly 2GB at about 90% of the verification process, causing the whole system to hang up.
This was tested on a 64-bit Ubuntu 14.04.1 system with 2GB RAM.

download's link not working

Could you please update the downloads link ?

The download link on the building page "https://github.com/libyal/libewf/wiki/Building" doesn't work.
Would like to download and build a stable copy.

Cheers.

ewfacquire: add functionality to support user suspend and resume of acquisition

handle leak on libewf_handle_close() ( version libewf-20150126 )

Brief:
After creating E01 image with libewf, i'm executing following code:

if(libewf_handle_close(ewfHandle, NULL) == -1)
     std::cout << "Can't close EWF file handle";
if(libewf_handle_free(&ewfHandle, NULL) == -1)
     std::cout << "Can't free EWF file handle";

Before libewf_handle_close i have, as expected, opened handle to %filename%.E01 file.
After execution of libewf_handle_free this code i have opened handle to %filename%.E02 file.

How i'm reproducing this:

1. I'm creating image of disk that is larger than destination disk i'm writing image too.
2. In this case output is:
   "Can't close EWF file handle"
   "Can't free EWF file handle"
3. On executing libewf_handle_close that is located inside libewf_handle_free (libewf_handle.c) *.E02 file is being created and handle to that file remains opened.

How i've fixed this
As long as libewf_handle_free calls libewf_handle_close, i left only call to libewf_handle_free this fixed that trouble - *.e01 handle is being closed, no other files are created. Memory is freed.

Trouble

1. In your sample (ewfacquire.c:2928) you are calling:

if( imaging_handle_close(
         ewfacquire_imaging_handle,
         &error ) != 0 )
    {
        fprintf(
         stderr,
         "Unable to close output file(s).\n" );

        goto on_error;
    }
if( imaging_handle_free(
       &ewfacquire_imaging_handle,
       &error ) != 1 )

imaging_handle_free after successfull call to imaging_handle_close. This might produce memory leak if imaging_handle_close will fail.

• If we call handle_free only after successful call to handle_close -> it might produce memory leak
• If we call handle_free always after handle_close -> this might produce handle leak (among with creating that *.e02 file)

libewf: add support to write EWF-L01 and EWF2-Lx01

ewfmount Cygwin fuse support

Look at adding Cygwin fuse support to ewfmount

pyewf

import pyewf
file_object = open("image.E01", "wb")

ewf_handle = pyewf.handle()

ewf_handle.open_file_objects([file_object])
ewf_handle.write("dddddd")

ewf_handle.close()

IOError: pyewf_handle_write_offset: unable to write offset. libewf_handle_write_offset: invalid handle - missing file IO pool

why?

libewf fails to build - libcstring_narrow_string_compare undefined

Hello,
libewf fails to build with following error:
libtool: link: gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -Wall -Wl,-z -Wl,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o .libs/ewfacquirestream byte_size_string.o digest_hash.o ewfacquirestream.o ewfinput.o ewfoutput.o guid.o imaging_handle.o log_handle.o platform.o process_status.o storage_media_buffer.o storage_media_buffer_queue.o -luuid -lcsystem -lcdatetime ../libewf/.libs/libewf.so -L/usr/lib64 -lcdata -lclocale -lcnotify -lcsplit -luna -lcfile -lcpath -lbfio -lfcache -lfdata -lfguid -lfvalue -lz -lbz2 -lhmac -lcaes -lcthreads -lcerror -lcstring
libtool: link: gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -Wall -Wl,-z -Wl,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o .libs/ewfacquire byte_size_string.o digest_hash.o device_handle.o ewfacquire.o ewfinput.o ewfoutput.o guid.o imaging_handle.o log_handle.o platform.o process_status.o storage_media_buffer.o storage_media_buffer_queue.o -lodraw -lsmdev -lsmraw -luuid -lcsystem -lcdatetime ../libewf/.libs/libewf.so -L/usr/lib64 -lcdata -lclocale -lcnotify -lcsplit -luna -lcfile -lcpath -lbfio -lfcache -lfdata -lfguid -lfvalue -lz -lbz2 -lhmac -lcaes -lcthreads -lcerror -lcstring
../libewf/.libs/libewf.so: undefined reference to `libcstring_narrow_string_compare'
collect2: error: ld returned 1 exit status
Makefile:1203: recipe for target 'ewfdebug' failed

I guess it is the same issue as patched in the pull request #46

The 'include

Best regards
Michal Ambroz

python binding for verify

I see that it is possible through check_file_signature_file() to see if an EWF file has checksums, but as far as I can tell there is no function for actually validating the checksum. This would be great, as it appears that the checksum is not equivalent to a checksum of the whole EWF file itself, but of a specific subset of the file.