velocidex/c-aff4

An AFF4 C++ implementation.

AFF4 -The Advanced Forensics File Format

The Advanced Forensics File Format 4 (AFF4) is an open source format used for the storage of digital evidence and data.

The standard is currently maintained here: https://github.com/aff4/Standard

Reference Images are found: https://github.com/aff4/ReferenceImages

This project implementats a C/C++ library for creating, reading and manipulating AFF4 images. The project also includes the canonical aff4imager binary which provides a general purpose standalone imaging tool.

The library and binary are known to work on Linux (all versions since Ubuntu 10.04), Windows (All versions) and OSX (All known versions).

What is currently supported.

Currently this library supports most of the features described in the standard https://github.com/aff4/Standard.

1. Reading and Writing ZipFile style volumes

   a. Supports splitting of output volumes into volume groups (e.g. splitting at 1GB volumes).

2. Reading ahd Writing Directory style volumes.

3. Reading and Writing AFF4 Image streams using the deflate or snappy compressor.

4. Reading RDF metadata using Turtle.

5. Multi-threaded imaging for efficient utilization on multi core systems.

What is not yet supported.

This implementation currently does not implement Section 6. Hashing of the standard. This includes verifying or generating linear or block hashes.

Copyright

Copyright 2015-2017 Google Inc. Copyright 2018-present Velocidex Innovations.

References

[1] "Extending the advanced forensic format to accommodate multiple data sources, logical evidence, arbitrary information and forensic workflow" M.I. Cohen, Simson Garfinkel and Bradley Schatz, digital investigation 6 (2009) S57–S68.

c-aff4's Issues

invalid LOC header - Zip64 location headers

Hello
When I try to read aff4 images created by winpmem tool, using aff4-java library, i get this error:

[ERROR] 00:04.393(Containers.java:getResourceID:136) invalid LOC header (bad signature)
java.util.zip.ZipException: invalid LOC header (bad signature)
at java.util.zip.ZipFile.read(Native Method)
at java.util.zip.ZipFile.access$1400(ZipFile.java:60)
at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:734)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.Reader.read(Reader.java:140)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2369)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2348)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2325)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2273)
at org.apache.commons.io.IOUtils.toString(IOUtils.java:1041)
at com.evimetry.aff4.Containers.getResourceID(Containers.java:132)
at com.evimetry.aff4.Containers.openContainer(Containers.java:91)
at com.evimetry.aff4.Containers.open(Containers.java:54)
at com.evimetry.aff4.examples.Information.main(Information.java:58)
[ERROR] 00:05.377(Information.java:main:95) org.apache.jena.atlas.RuntimeIOException: java.nio.channels.ClosedChannelException

I have opened an issue in this library, and I was told, that is caused by that winpmem's zip implementation is generating invalid Zip64 location headers.

Also i tried to verify this images by evimetry collector, but I also get an error:

Error: Unable to verify file Memory-3.1.aff4 due to error:
The first file entry in the container is missing file length information

Could you fix that or propose a solution how to read this aff4 images using java?

docker built winpmem.exe is missing the driver

This is my bad. I removed the strip hook from winpmem's Makefile.am, and opted for using make install-strip. Apparently this strips out the driver resource that's plunked onto the end of the file.

Create example that enumerates a disk image

I have a long-standing issue on bulk_extractor to add AFF4 support. To do this, I need an example that allows me to enumerate all of the bytes in an AFF4 volume.

• Is there an example of how to do this?
• If not, is this worth doing?

Logical imaging issue using find - flat file output only

System: Linux Mint 18 Cinnamon 64-bit
Kernel: Linux 4.4.0-119-generic

I have a test directory (.../testdir/...) located in the same directory as linpmem. Running this command:
find ./testdir -type f
...results in this:
./testdir/subdir1/.hiddendir1/.hiddendir11/fileh111.txt ./testdir/subdir1/.hiddendir1/fileh11.txt ./testdir/subdir1/.hiddendir1/subdir11/file111.txt ./testdir/subdir1/file1.txt ./testdir/subdir2/.hiddendir2/subdir21/file211.txt ./testdir/subdir2/.hiddendir2/fileh21.txt ./testdir/subdir2/.hiddendir2/.hiddendir21/fileh211.txt ./testdir/subdir2/file2.txt ./testdir/file.txt ./testdir/subdir3/file3.txt ./testdir/subdir3/.hiddendir3/fileh31.txt ./testdir/subdir3/.hiddendir3/subdir31/file311.txt ./testdir/subdir3/.hiddendir3/.hiddendir31/fileh311.txt
...which is correct.

Running this command:
find ./testdir -type f | linpmem -i @ -o ./img_logical.aff4
...results in a flat text file with zero bytes named "img_logical.aff4":
$ shed --hex img_logical.aff4 shed: img_logical.aff4 has zero size

"make install" keeps appending driver to winpmem.exe

I noticed that the driver is being appended to winpmem.exe each time that "make install" is called in tools/pmem. The binary will continue to grow and grow. It appears that the "all-local" target, where the appending of the driver occurs, is run each time a "make install" is run. No checking is done to see if the driver has already been appended.

As mentioned here, authenticode signing will not cover the appended driver, so it may be a good idea to think of other approaches. IIRC, winpmem 2.x included the drivers as a resource, which is included as part of an authenticode signature. Is there a good reason that this can't be done, again?

Maps with only a single entry are invalid

Single entry maps use an old broken layout for the map entry. See line

c-aff4/aff4/aff4_map.cc

Line 569 in 1c463e8

AddRange(0, last_target->Size(), last_target->Size(), last_target);

The signature is AddRange(map_offset, target_offset, length, *target)

It should read: AddRange(0, 0, last_target->Size(), last_target);

pmem with "--output -" dumps to a file named "-" rather than writing to STDOUT

Just noticed this when I was working on testing winpmem out, some more. Where it used to be possible to dump to STDOUT with --output - under rc3, the current master (rc8) instead dumps memory to a file named "-". I haven't had the time to track down why this is happening, but one of the things I'm working on needs this capability.

zlib compression is not in the AFF4 standard

I noticed in commit e5e1489 that you re-added the option to aff4imager to support the "zlib" compression (deflate is supported). @blschatz pointed out to me that this form of compression isn't technically part of the AFF4 standard. I don't think we should allow options to create non-standards compliant images, as it's going to make thing worse for tool vendors who want to read the images.

streaming to stdout on Windows fails (winpmem)

Related to issue #8

Out of curiosity, did you try to open the resulting file that was streamed to stdout ? For example, this fails on a Windows 10 machine:

>winpmem_3.0.rc1.exe --format raw --output - > test.zip
2018-08-27 11:54:33 E Local file header invalid!

Looking at the resulting file:

$ ls -lh test.zip 
-rwxrwxrwx  1 jamie 1552576430   2.1G Aug 27 10:54 test.zip

$ unzip -l test.zip 
Archive:  test.zip
aff4://add12264-5091-41c4-8f64-e2530726291f
error [test.zip]:  missing 6351011913 bytes in zipfile
  (attempting to process anyway)
error [test.zip]:  attempt to seek before beginning of zipfile
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)

$ 7z l test.zip 

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)

Scanning the drive for archives:
1 file, 2238923532 bytes (2136 MiB)

Listing archive: test.zip


ERROR: test.zip : opening : E_FAIL

Errors: 1


System ERROR:
E_FAIL

$ zip -FFv test.zip --out fixed.zip
Fix archive (-FF) - salvage what can
 Found end record (EOCDR) - says expect single disk archive
  Found archive comment
Scanning for entries...
 Local ( 1      0): copying: container.description  .....................................................................................................................................................................................................................
	zip warning: no end of stream entry found: container.description
	zip warning: rewinding and scanning for later entries
	zip warning: zip file empty

From what I can gather, it's failing in aff4/zip.cc and just stops with: file_header.compression_method == 135 and zip_info->compression_method == 0

Also: file_header_filename == "" and zip_info->filename == container.description

osxpmem no longer compiles

It seems that the version of osxpmem in this repo hasn't been updated in a while and doesn't support the framework changes.

I think the pmem tools should be removed from this repo and given their own home as they're not really relevant to libaff4 as a product.

winpmem_v3.3.rc3.exe Windows 10 Raw output

Command: winpmem_v3.3.rc3.exe --formate raw --output memory.dmp
Computer: Windows 10 Enterprise Build 17134 (8gb Ram)

When I start to use Volatility ImageInfo and KDBGscan don't find anything to go off.
Suggested Profiles : No Suggestion (Instantiated with Win10x64_15063).

I have tried a few different versions of volatility so I suspect the way the memory dump is being collected is at fault. Any ideas?

Build fails with spdlog 1.1.0

Hello, building the latest c-aff4 c5c96e6 fails if spdlog 1.1.0 is installed:

libaff4.cc:554:24: error: no member named 'stderr_color_mt' in namespace 'spdlog'
        return spdlog::stderr_color_mt("aff4");
               ~~~~~~~~^
1 error generated.
make[2]: *** [libaff4.lo] Error 1

ccccccijfdbnelgjulibhjreenidkkdciieufhrjltej

make install fails if DESTDIR is used

If I install with make install DESTDIR=/somewhere, then it fails with:

make[3]: Entering directory `.../c-aff4-fd984ea3cddc034740d5d03ba510a407a2e451a3/tools/pmem'
Creating deployable bundle.
rm -rf osxpmem.app osxpmem.zip
mkdir osxpmem.app/
cp README.md /opt/local/bin/osxpmem osxpmem.app/  || echo You must run make install before building a bundle.
cp: /opt/local/bin/osxpmem: No such file or directory
You must run make install before building a bundle.
dylibbundler -x osxpmem.app/osxpmem -b -d osxpmem.app/libs/ -p @executable_path/libs/ -of -od -cd
* Collecting dependencies/opt/local/libexec/llvm-5.0/bin/llvm-objdump: 'osxpmem.app/osxpmem': No such file or directory
fatal error: otool: internal objdump command failed
Cannot find file osxpmem.app/osxpmem to read its dependencies
make[3]: *** [install-exec-hook] Error 1

I don't have any need for a relocatable app version of osxpmem, and even if this were to succeed (because /opt/local/bin/osxpmem already existed because I already had c-aff4 installed), the resulting osxpmem.app isn't actually installed anywhere, making this code wholly pointless in make install. For the MacPorts aff4 port, I have removed the entire install-exec-hook block from tools/pmem/Makefile.am. Offer a make bundle target, if you like (updated to support DESTDIR), but it shouldn't be a required part of make install.

Image stream contains a sub-chunk size chunk

All chunks in the image stream should have an uncompressed size of chunkSize. In some instances libaff4 writes the last chunk without padding the uncompressed data to chunkSize. Suspect the problem is with flush().

This is important as decompressors (exp LZ4) are faster if they know the size of the decompressed buffer.

winpmem_3.1.rc3.exe does not work correctly on Windows 7 or 10

My organization is trying to use the latest winpmem_3.1.rc3.exe for the VSM support, but it isn't working for us at all, even without VSM and on Windows 7. On every machine I have tried (all 64-bit, various Windows10 releases - 1803, 1810, 1709) the resulting image is far, far larger than physical memory (tried 8GB and 16GB) - ultimately I've had to kill the capture before the hard drive fills up. We've tried it with --format raw, --format elf, and with --output to stdout and a file.

Stored ImageStreams are invalid

When not using compression, libaff4 generates the wrong metadata for the relevant ImageStream. Eg:

aff4:compressionMethod aff4:NullCompressor ;

In this case it should not store a compressionMethod. From the spec v1

"Where compression is used within the Image Stream, the object must have a property aff4:compressionMethod set to a resource identifying the compression algorithm. Where there is no compressionMethod set, it is assumed that chunks are stored."

Logical Imaging - File Descriptor Limits

System: Linux Mint 18 Cinnamon 64-bit
Kernel: Linux 4.4.0-119-generic
RAM: 15.6 GiB
linpmem: 3.0.rc1

I ran out of file descriptors pretty quickly. To image a ~1 GB home directory, I obtained about 134 MB of artifacts. This is a typical home directory on Linux Mint 18 with a typical degree of nesting, so nothing edge-case unusual. Here's the output from my application:

Image user's home directory? [y/n]
y
Starting home directory imaging (may take several minutes!)...
/usr/bin/find: ‘/home/hharness/.cache/dconf’: Permission denied
2018-04-20 08:40:02 E Can not open file /home/hharness/.config/google-chrome/Default/File
System/006/p/03/00000343: Too many open files
...

The permissions error was due to not running as root and not something I'm worried about here. The "Too many open files" warnings continue for about 4 MB of text file. I had less trouble imaging /var/log with only errors related to permissions. The system I tested on has a home directory file count of 13,235 (find ~ -type f) and a file-max of 1,609,982 (cat /proc/sys/fs/file-max). Immediately after a reboot, it shows 51,335 open files (lsof | wc -l). That's a lot of open file descriptors available for a typical home directory that only has 13,235 available to find -type f. I can send you the text file of the screen messages if you want (4 MB in size and I'd rather not make it public just yet).

I may be misinterpreting my results from "cat ...file-max" vs "lsof ...wc" as well as the "Too many open files" error. I re-ran the test this way (outside my program) with the same results except those with previous permissions errors weren't excluded because of that this time:

$ sudo /usr/bin/find ~ -type f | ./thirdparty/linpmem -i @ -o /media/hharness/EVIDENCE/test_lin/img_logical.aff4

I'm sure I'm running the correct version of linpmem now:
$ ./linpmem --version

./linpmem version: 3.0.rc1

It's bound to be something other than file descriptor allocation. I expect at ~16 GB of RAM, that's not the issue, either.

Extracting PhysicalMemory from AFF4

I have created a multi-stream dump with winpmem-2.1.post4.exe

I try to work with the aff4 with the latest release winpmem_v3.3.rc3.exe
This is what I have as a result:

>winpmem_v3.3.rc3.exe d:\dump.aff4 -V
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix aff4: <http://aff4.org/Schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix memory: <http://aff4.org/Schema#memory/> .

<aff4://f12af4c6-58e3-44d2-a79a-927311885ef8/PhysicalMemory>
    aff4:category memory:physical ;
    aff4:stored <aff4://f12af4c6-58e3-44d2-a79a-927311885ef8> ;
    a aff4:map .

<aff4://f12af4c6-58e3-44d2-a79a-927311885ef8/PhysicalMemory/data>
    aff4:chunk_size 32768 ;
    aff4:chunks_per_segment 1024 ;
    aff4:compression <https://www.ietf.org/rfc/rfc1950.txt> ;
    aff4:size 16902103040 ;
    aff4:stored <aff4://f12af4c6-58e3-44d2-a79a-927311885ef8> ;
    a aff4:image .

<aff4://f12af4c6-58e3-44d2-a79a-927311885ef8/c:/pagefile.sys>
    aff4:category memory:pagefile ;
    aff4:chunk_size 32768 ;
    aff4:chunks_per_segment 1024 ;
    aff4:compression <https://www.ietf.org/rfc/rfc1950.txt> ;
    memory:pagefile_number 0 ;
    aff4:size 25688113152 ;
    aff4:stored <aff4://f12af4c6-58e3-44d2-a79a-927311885ef8> ;
    a aff4:image .

When I try to extract the PhysicalMemory, I get the following error:

>winpmem_v3.3.rc3.exe -e PhysicalMemory -o memory.raw d:\dump.aff4
2019-12-11 16:34:44 E Cannot specify an export and an output volume at the same time (did you mean --export_dir).
2019-12-11 16:34:44 E Imaging failed with error: INVALID_INPUT

I've read a similar issue (#46), and tried your suggestion, but without any result.

>winpmem_v3.3.rc3.exe -e "*/PhysicalMemory" -D test d:\dump.aff4
>

When I try to export with the older winpmem-version, it seems to work correctly

>winpmem-2.1.post4.exe -e PhysicalMemory -o memory.raw d:\dump.aff4
Extracting aff4://f12af4c6-58e3-44d2-a79a-927311885ef8/PhysicalMemory into file:///c:/memory.raw
 Reading 0x8000  0MiB / 18376MiB 0MiB/s
 Reading 0xb50000  11MiB / 18376MiB 41MiB/s
 Reading 0x16d8000  22MiB / 18376MiB 42MiB/s
 Reading 0x2320000  35MiB / 18376MiB 45MiB/s
 Reading 0x2d90000  45MiB / 18376MiB 41MiB/s
 Reading 0x3890000  56MiB / 18376MiB 41MiB/s
 Reading 0x42f8000  66MiB / 18376MiB 39MiB/s
 Reading 0x4db0000  77MiB / 18376MiB 41MiB/s
 Reading 0x5950000  89MiB / 18376MiB 45MiB/s
...

Am I missing a step ? Or is it something else?
Thank you

Winpmem BSOD on VSM enabled systems

This has been reported a while back
https://df-stream.com/2017/08/memory-acquisition-and-virtual-secure/

Unfortunately for the moment we can not fix it without rebuilding the driver and this requires a new code signing cert.

winpmem 3.1rc3 - dumping in "raw" format never completes on Windows, output file grows until no disk space left

On Windows, attempts to use winpmem 3.1rc3 to dump memory in "raw" format never completes, and the output file created grows so large (hundreds of gigabytes in size) that the disk runs out of space. I've narrowed this down to only happening when the system has memory ranges with starting addresses >= 0x100000000 (2^32). When the memory ranges are confined to < 0x100000000, raw memory dumps succeed.

On Windows 7, I found these ranges would start showing up when I configured the system to have >= 3584MB (3.5GB) of RAM. At 3583MB of RAM, raw memory dumping would succeed.

The command I'm using in all cases is:

winpmem_3.1.rc3.exe --format raw --output raw.dump

Here are runs that illustrate the problem that I am encountering

• Windows 7 64 bit, 3583MB of RAM:
  • Memory range start addresses: 0x00001000, 0x00100000
  • raw memory dump completes
  • Command output
• Windows 7 64 bit, 3584MB of RAM
  • Memory range start addresses: 0x00001000, 0x00100000, 0x100000000
  • This seems to be the magic amount of memory on Windows 7 for the 0x100000000 memory address to start being used.
  • raw memory dump never completes, output file is hundreds of gigabytes in size, no disk space left.
  • Command output
• Windows 7 64 bit, 4096MB of RAM
  • Memory range start addresses: 0x00001000, 0x00100000, 0x100000000
  • raw memory dump never completes, output file is hundreds of gigabytes in size, no disk space left.
  • Command output
• Windows 10 64 bit, 16GB of RAM
  • Memory range start addresses: 0x00001000, 0x00100000, 0x00103000, 0x100000000
  • raw memory dump never completes, output file is hundreds of gigabytes in size, no disk space left.
  • Command output

GLIBC 2.25 Dependency

System: Linux Mint 18.3 Cinnamon 64-bit
Kernel: Linux 4.13.0-39-generic
RAM: 15.6 GiB
linpmem_3.0.rc1.bin (from Release 1.0 RC2)

It looks like the latest RC2 builds were against GLIBC 2.25. My test machine is running the latest Mint 18.3 (4.13 kernel) and the repos are only up to GLIBC 2.23. Upgrading GLIBC on a workstation is a risky proposition and not really an option on a live suspect machine. Here's my output:

$ sudo /usr/bin/find ~ -type f | ./thirdparty/linpmem -i @ -o /media/hharness/EVIDENCE/test_lin2/img_logical.aff4
[sudo] password for hharness: [...]
./thirdparty/linpmem: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by ./thirdparty/linpmem)

$ ldd --version
ldd (Ubuntu GLIBC 2.23-0ubuntu10) 2.23
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

$ /lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Ubuntu GLIBC 2.23-0ubuntu10) stable release version 2.23, by Roland McGrath et al.
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 5.4.0 20160609.
Available extensions:
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs.

[eof]

Stack corruption on invalid input leading to exploitable conditions

I guess my yearly bug on aff4 ;) j/k... I was just looking to see how similar this was to that last one. Not entirely.

This is from winpmem 2.1.post4, I figured you might see it better over here, but there's also a number of other bugs in winpmem I'll try to file them all soon I've been busy and this had been annoying me for a while. An attacker can prevent the use of winpmem by setting the temp environment variable to a long value. TEMP or TMP to TEMP=c:\AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA will cause this, the directory has to exist (I think ;).

01 007de530 759d5bee 00000003 007de584 759d6165 KERNEL32!ExitProcess+0x13
02 007de53c 759d6165 00000003 1a9ba4b8 00587882 msvcrt!exit+0x8e
03 007de584 759d5b71 00000003 00000001 00000000 msvcrt!initterm_e+0x1c5
04 007de598 759cb349 00000003 00010001 007de5f4 msvcrt!exit+0x11
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Users\files\Downloads\winpmem-2.1.post4.exe - 
05 007de8c8 004971a8 00000001 00496f86 00000000 msvcrt!abort+0xf9
06 007de8e8 004924fb 6f4736b4 fffffffe 007de9d8 winpmem_2_1_post4!pcre_free_substring+0x33739
07 007de8f8 00492447 00000002 0059f748 00000023 winpmem_2_1_post4!pcre_free_substring+0x2ea8c
08 007de9d8 00491e1d 005ad3fc 03130fe0 007dea08 winpmem_2_1_post4!pcre_free_substring+0x2e9d8
09 007dea58 004951f3 007dea78 00576920 0057bc58 winpmem_2_1_post4!pcre_free_substring+0x2e3ae
0a 007deac8 00406ee7 005bbf60 00587dd0 0313b110 winpmem_2_1_post4!pcre_free_substring+0x31784
0b 007deba8 00407504 007dfbf0 007dfbe0 00000001 winpmem_2_1_post4+0x6ee7
0c 007dfc88 00408a51 007dfcfc 007dfd03 007dfd01 winpmem_2_1_post4+0x7504
0d 007dfd28 00413459 00000002 03130c50 007dfdec winpmem_2_1_post4+0x8a51
0e 007dfe28 00403763 00000002 03130c50 007dffcc winpmem_2_1_post4+0x13459
0f 007dfea8 004013e0 03138fb0 00000003 00000002 winpmem_2_1_post4+0x3763
10 007dff80 761b8654 0028f000 761b8630 0058ddcb winpmem_2_1_post4+0x13e0
11 007dff94 76f34a47 0028f000 3972b912 00000000 KERNEL32!BaseThreadInitThunk+0x24
12 007dffdc 76f34a17 ffffffff 76f59ed9 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x137

E0309 08:57:43.932703  8248 aff4_file.cc:138] Can not open file :The filename, directory name, or volume label syntax is incorrect.
F0309 08:57:43.932703  8248 win_pmem.cc:422] Unable to create driver file.
*** Check failure stack trace: ***
   @   004924fb
   @   00492447
   @   00491e1d
   @   004951f3
   @   00406ee7
   @   00407504
   @   00408a51
   @   00413459
   @   00403763
   @   004013e0
   @   761b8654
   @   76f34a47
   @   76f34a17

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

winpmem 1.6.2 does not crash the same way but has other errors.

If I send a PR or some more issues is the 2.x series the mainline now?
K2
P.S. go check out my network sha256 server for memory integrity. I'll port my volatility module soon as we get these fixed ;)

Multiple map problems

Using winpmem v3.3.rc1, we created a memory image of an arch linux vm.

The idx file references a non-existent stream. (suspect the idx file has been truncated or not fully written correctly).
The idx file has multiple duplicate entries.
The map references the non-existent stream.

The map idx looked as follows:
0 => aff4://52470487-b665-46e6-97de-071325d32dbd/PhysicalMemory/data
1 => aff4://52470487-b665-46e6-97de-071325d32dbd/PhysicalMemory/data
2 => aff4://52470487-b665-46e6-97de-071325

---

0 => aff4://52470487-b665-46e6-97de-071325d32dbd/PhysicalMemory/data
1 => aff4://52470487-b665-46e6-97de-071325d32dbd/PhysicalMemory/data
2 => aff4://52470487-b665-46e6-97de-071325
[0x00000000:0x00001000] => SPARSE
[0x00001000:0x0009e000] => stream 1 [0x00000000:0x0009e000]
[0x0009f000:0x00061000] => SPARSE
[0x00100000:0x00300000] => stream 1 [0x0009e000:0x00300000]
[0x00400000:0x00001000] => stream 2 [0x00000000:0x00001000]
[0x00401000:0x00bfe000] => stream 1 [0x0039e000:0x00bfe000]
[0x00fff000:0x00001000] => SPARSE
[0x01000000:0x00001000] => stream 2 [0x00000000:0x00001000]
[0x01001000:0x003ff000] => stream 1 [0x00f9c000:0x003ff000]
[0x01400000:0x00001000] => stream 2 [0x00001000:0x00001000]
[0x01401000:0x3ebef000] => stream 1 [0x0139b000:0x3ebef000]

information.yaml:

Imager: WinPmem 3.3rc1
Registers:
CR3: 114032640
NtBuildNumber: 2600
KernBase: 2152558592
NtBuildNumberAddr: 2153066728
Runs:

• start: 4096
  length: 647168
• start: 1048576
  length: 15724544
• start: 16777216
  length: 1056899072

winpmem.exe memory acquisition errors out

Greetings--

I'm running into the error below while attempting to acquire a memory sample with winpmem.exe (1.0rc1 build). I'm executing it on Windows 10 Pro (Version 1709 16299.371). While attempting to reproduce this issue on another system, I experienced the same thing on a Windows 10 Enterprise image.

Standard output:

C:\Users\Adam\Desktop>.\winpmem.exe -o C:\Users\Adam\Desktop\test.aff4
Driver Unloaded.
2018-04-22 18:27:18 E Cant create directory \\: The filename, directory name, or volume label syntax is incorrect.

2018-04-22 18:27:18 E Unable to create intermediate directory: The operation completed successfully.

2018-04-22 18:27:18 E Unable to open device: The operation completed successfully.

2018-04-22 18:27:18 E Imaging failed with error: -8
Driver Unloaded.

Debug output:

C:\Users\Adam\Desktop>.\winpmem.exe -ddd -o C:\Users\Adam\Desktop\test.aff4
2018-04-22 19:35:54 I This is The WinPmem memory imager. version 2.1.post4
2018-04-22 19:35:54 I Extracting WinPmem drivers from binary.
2018-04-22 19:35:54 D AFF4FactoryOpen : aff4://9b6f80ef-e329-432e-86f4-8329f992c748
2018-04-22 19:35:54 D AFF4FactoryOpen : file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:54 D Opening file C:\Users\Adam\Desktop\winpmem.exe
2018-04-22 19:35:54 D AFF4FactoryOpen (new instance): file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:54 D Found ECD at 39117b
2018-04-22 19:35:54 I Loaded AFF4 volume URN aff4://81c68412-ab86-42b0-a456-8e6340b5671b from zip file.
2018-04-22 19:35:54 I Global offset: 266800
2018-04-22 19:35:54 D Found file container.description @ 12a558
2018-04-22 19:35:54 D Found file version.txt @ 7e
2018-04-22 19:35:54 D Found file information.turtle @ 12a5d6
2018-04-22 19:35:54 D Found file fcat.exe @ e8
2018-04-22 19:35:54 D Found file winpmem_x86.sys @ 120003
2018-04-22 19:35:54 D Found file winpmem_x64.sys @ 11652e
2018-04-22 19:35:54 D Returning: file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:54 D AFF4FactoryOpen : file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:54 D AFF4FactoryOpen (cached): file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:54 D Returning: file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:54 D Opening ZipFileSegment aff4://81c68412-ab86-42b0-a456-8e6340b5671b/information.turtle
2018-04-22 19:35:54 D Returning: aff4://81c68412-ab86-42b0-a456-8e6340b5671b/information.turtle
2018-04-22 19:35:54 D AFF4FactoryOpen (new instance): aff4://81c68412-ab86-42b0-a456-8e6340b5671b
2018-04-22 19:35:54 I Openning driver AFF4 volume: aff4://81c68412-ab86-42b0-a456-8e6340b5671b
2018-04-22 19:35:54 D Returning: aff4://81c68412-ab86-42b0-a456-8e6340b5671b
2018-04-22 19:35:54 I Temp path C:\Users\Adam\AppData\Local\Temp
2018-04-22 19:35:54 D AFF4FactoryOpen : aff4://81c68412-ab86-42b0-a456-8e6340b5671b/winpmem_x64.sys
2018-04-22 19:35:54 D AFF4FactoryOpen : aff4://81c68412-ab86-42b0-a456-8e6340b5671b
2018-04-22 19:35:54 D AFF4FactoryOpen (cached): aff4://81c68412-ab86-42b0-a456-8e6340b5671b
2018-04-22 19:35:54 D AFF4FactoryOpen : file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:54 D AFF4FactoryOpen (cached): file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:54 D Returning: file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:54 D Returning: aff4://81c68412-ab86-42b0-a456-8e6340b5671b
2018-04-22 19:35:54 D AFF4FactoryOpen (new instance): aff4://81c68412-ab86-42b0-a456-8e6340b5671b/winpmem_x64.sys
2018-04-22 19:35:54 D AFF4FactoryOpen : file:///C:/Users/Adam/AppData/Local/Temp/pme2A7E.tmp
2018-04-22 19:35:54 D Opening file C:\Users\Adam\AppData\Local\Temp\pme2A7E.tmp
2018-04-22 19:35:54 D Creating intermediate directories C:\
2018-04-22 19:35:54 D Creating intermediate directories C:\Users\
2018-04-22 19:35:54 D Creating intermediate directories C:\Users\Adam\
2018-04-22 19:35:54 D Creating intermediate directories C:\Users\Adam\AppData\
2018-04-22 19:35:54 D Creating intermediate directories C:\Users\Adam\AppData\Local\
2018-04-22 19:35:54 D Creating intermediate directories C:\Users\Adam\AppData\Local\Temp\
2018-04-22 19:35:54 D AFF4FactoryOpen (new instance): file:///C:/Users/Adam/AppData/Local/Temp/pme2A7E.tmp
2018-04-22 19:35:54 I Extracted aff4://81c68412-ab86-42b0-a456-8e6340b5671b/winpmem_x64.sys to file:///C:/Users/Adam/AppData/Local/Temp/pme2A7E.tmp
2018-04-22 19:35:55 D AFF4FactoryOpen : file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:55 D AFF4FactoryOpen (cached): file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:55 D Returning: file:///C:/Users/Adam/Desktop/winpmem.exe
2018-04-22 19:35:55 D Closing object file:///C:/Users/Adam/AppData/Local/Temp/pme2A7E.tmp 0
2018-04-22 19:35:55 D Returning: aff4://81c68412-ab86-42b0-a456-8e6340b5671b/winpmem_x64.sys
Driver Unloaded.
2018-04-22 19:35:55 I Loaded Driver C:\Users\Adam\AppData\Local\Temp\pme2A7E.tmp
2018-04-22 19:35:55 D AFF4FactoryOpen : file://./pmem
2018-04-22 19:35:55 D Opening file \\.\pmem
2018-04-22 19:35:55 D Creating intermediate directories \
2018-04-22 19:35:55 D Creating intermediate directories \\
2018-04-22 19:35:55 I MkDir \\
2018-04-22 19:35:55 E Cant create directory \\: The filename, directory name, or volume label syntax is incorrect.

2018-04-22 19:35:55 E Unable to create intermediate directory: The filename, directory name, or volume label syntax is incorrect.

2018-04-22 19:35:55 E Unable to open device: The filename, directory name, or volume label syntax is incorrect.

2018-04-22 19:35:55 E Imaging failed with error: -8
Driver Unloaded.
2018-04-22 19:35:55 I Removed C:\Users\Adam\AppData\Local\Temp\pme2A7E.tmp

Extracting PhysicalMemory from aff4 format

How do I extract raw memory data from aff4 using pmem 3.x?

Whta I tried is here:

> winpmem_3.1.rc6.exe -o win10x86.aff4

Succeed to create memory dump with aff4 format

> winpmem_3.1.rc6.exe -V win10x86.aff4
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix aff4: <http://aff4.org/Schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix memory: <http://aff4.org/Schema#memory/> .

<aff4://aba09986-2a6b-40e8-9feb-3af9b27137e1/PhysicalMemory>
    aff4:category memory:physical ;
    aff4:stored <aff4://aba09986-2a6b-40e8-9feb-3af9b27137e1> ;
    a aff4:Map .

<aff4://aba09986-2a6b-40e8-9feb-3af9b27137e1/PhysicalMemory/data>
    aff4:chunkSize 32768 ;
    aff4:chunksInSegment 1024 ;
    aff4:compressionMethod <https://www.ietf.org/rfc/rfc1950.txt> ;
    aff4:size 1073205248 ;
    aff4:stored <aff4://aba09986-2a6b-40e8-9feb-3af9b27137e1> ;
    a aff4:ImageStream .
...(snip)

> winpmem_3.1.rc6.exe -e PhysicalMemory -o win10x86.raw win10x86.aff4
2018-10-30 22:49:39 E Can not specify an export and an output volume at the same time (did you mean --export_dir).
2018-10-30 22:49:39 E Imaging failed with error: -5

> winpmem_3.1.rc6.exe -e PhysicalMemory --export_dir test win10x86.aff4
> winpmem_3.1.rc6.exe -e PhysicalMemory -D test win10x86.aff4

Both commands returns no message and creates no files

winpmem-2.1.post4.exe works -e and -o option to extract stream data. It was very useful for me.

PhysicalMemory Extraction using linpmem

Our team completed a memory collection using winpmem version 2.1 on an Azure VM running Windows Server 2016
The output was in AFF4 format.

I am attempting to extract the PhysicalMemory for processing with Rekall and Volatility and receiving the following output results.

root@workstation:# ./linpmem-v3.3.rc1 -V ../winpmem_collection.aff4
@Prefix rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns# .
@Prefix aff4: http://aff4.org/Schema# .
@Prefix xsd: http://www.w3.org/2001/XMLSchema# .
@Prefix memory: http://aff4.org/Schema#memory/ .

aff4://90283ff6-6777-4497-ac03-c711a870c8c0/PhysicalMemory
aff4:category memory:physical ;
aff4:stored aff4://90283ff6-6777-4497-ac03-c711a870c8c0 ;
a aff4:map .

aff4://90283ff6-6777-4497-ac03-c711a870c8c0/PhysicalMemory/data
aff4:chunk_size 32768 ;
aff4:chunks_per_segment 1024 ;
aff4:compression https://www.ietf.org/rfc/rfc1950.txt ;
aff4:size 34359267328 ;
aff4:stored aff4://90283ff6-6777-4497-ac03-c711a870c8c0 ;
a aff4:image .

file:///root/winpmem_collection.aff4
aff4:contains aff4://90283ff6-6777-4497-ac03-c711a870c8c0 .

root@workstation:#

root@workstation:# ./linpmem-v3.3.rc1 -e '/PhysicalMemory' -D /root/linpmem/ ../winpmem_collection.aff4
root@workstation:# ll
total 2724
drwxr-xr-x 2 root root 4096 Feb 21 19:49 ./
drwx------ 9 root root 4096 Feb 21 20:03 ../
-rwxrwxrwx 1 root root 2779456 Feb 19 15:37 linpmem-v3.3.rc1
root@workstation:# pwd
/root/linpmem
root@workstation:#

root@workstation:# ./linpmem-v3.3.rc1 -e '*/PhysicalMemory' --volume_format raw --output /root/linpmem/mem.raw ../winpmem_collection.aff4
2020-02-21 20:08:14 E Cannot specify an export and an output volume at the same time (did you mean --export_dir).
2020-02-21 20:08:14 E Imaging failed with error: INVALID_INPUT
root@workstation:#

snappy compression not supported in linux 3.3rc1

Im extracting physical memory collected with snappy compression and it looks as though this compression method isnt implemented in the linux binary.

Collection: winpmem_v3.3.rc3.exe -dd -o yolo.aff -t -c snappy

Extraction linux (3.3rc1:
./linpmem-v3.3.rc1 -e */PhysicalMemory -D output/ yolo.aff
2019-12-19 06:22:58 E Compression method https://tools.ietf.org/html/rfc1951 is not supported by this implementation.
NOT_IMPLEMENTED: at volume_group.cc: 29
NOT_IMPLEMENTED: at aff4_map.cc: 102
NOT_IMPLEMENTED: at volume_group.cc: 65
NOT_IMPLEMENTED: at aff4_imager_utils.cc: 28
2019-12-19 06:22:58 E Error: NOT_IMPLEMENTED

I am able to extract with the windows rc3:
winpmem_v3.3.rc3.exe -dd -e */PhysicalMemory -D output yolo.aff4

C API has no way to control logigng or return error messages

The C API has no way to control logging or return error messages. It would be a great improvement if instead of having spdlog print to stderr unconditionally, that those messages were collected for use by the caller of libaff4 instead.

I'm willing to submit a patch for this if the libaff4 devs are agreeable.

rdf_tests.cc fails to compile

rdf_tests.cc:16:40: error: ‘class aff4::URN’ has no member named ‘Parse’
   uri_components components = URN(url).Parse();
                                        ^~~~~
rdf_tests.cc:61:43: error: ‘class aff4::URN’ has no member named ‘Parse’
   components = URN("http:www.google.com").Parse();
                                           ^~~~~

So far as I can see, there's no Parse() in URN or any of its superclasses.

Local file header is invalid

At present the ZIP files produced by c-aff4 are divergent from the ZIP specification - a Data Descriptor should follow the archive content when bit 3 is set in the Local File Header. While the infozip utilities appear forgiving of violations of this subtlety of the standard, other implementations aren't (e.g. 010 editor ZIP parsers, Evimetry, and aff4-cpp-lite for a start).

THE REASON THIS IS IMPORTANT: This part of the ZIP spec allows for a ZIP implementation to start streaming content to a ZIP archive segment, without knowing what size the file with be up front. The file header uses bit 3 of the Local File Header to indicate that the size information will be supplied later, when all the content was been written into the archive. The size is then specified using the Data Descriptor.

With this current bug, if an AFF4 file is truncated, and the Central Directories are lost, it is difficult to carve out the individual files of the corrupt ZIP, as the compressed and uncompressed sizes are missing.

Some useful background info: adamhathcock/sharpcompress#88

winpmem.exe map dumping uses a wildly fluctuating amount of memory

While executing winpmem.exe --format map --output foo.aff4, I noticed that the memory utilization jumps up and down between 20-50 MB of RAM. When dumping in raw mode, winpmem.exe tends to sit right around ~1.4MB of RAM, only fluctuating +/- 20 KB.

The wild fluctuation appears to be due, in part, to AFF4Status WinPmemImager::WriteMapObject_() re-declaring of a 1MB buffer over and over again. This is compounded upon by the design of the AFF4Stream::Read interface: virtual std::string Read(size_t length);, which requires any implementation to create and return a new buffer. Naive implementations of AFF4::ReadBuffer simply call auto result = Read(*length);, causing an additional allocation of a potentially large buffer.

I have the following observations:

• The constant allocation/deallocation of memory increases the chances of forensic data loss.
• The 1MB buffer as used by WriteMapObject_, to me, seems overly large. When it comes to writing to disk, it has been my experience that there's little benefit to a buffer size beyond ~80KB.
• When it comes to data buffers, it has been my experience that they should be caller-allocated. The caller is then left with the option of re-using the same buffer for subsequent calls, which are likely to occur when reading large quantities of data. This would mean eliminating AFF4Stream::Read in favor of AFF4Stream::ReadBuffer.

pmem command options

Would be great if winpmem (and the others) can automatically run when you execute them without command line arguments and dump memory to the folder that the executable runs from (similar to how dumpit operates)

ZipTest fails and aff4-test crashes

[juckelman@midas tests]$ ./aff4-test 
[==========] Running 21 tests from 10 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from MemoryDataStoreTest
[ RUN      ] MemoryDataStoreTest.IncompatibleGet
[       OK ] MemoryDataStoreTest.IncompatibleGet (4 ms)
[ RUN      ] MemoryDataStoreTest.StorageTest
[       OK ] MemoryDataStoreTest.StorageTest (3 ms)
[ RUN      ] MemoryDataStoreTest.TurtleSerializationTest
[       OK ] MemoryDataStoreTest.TurtleSerializationTest (7 ms)
[----------] 3 tests from MemoryDataStoreTest (14 ms total)

[----------] 1 test from AFF4ObjectCacheTest
[ RUN      ] AFF4ObjectCacheTest.TestLRU
[       OK ] AFF4ObjectCacheTest.TestLRU (3 ms)
[----------] 1 test from AFF4ObjectCacheTest (3 ms total)

[----------] 1 test from StreamTest
[ RUN      ] StreamTest.StringIOTest
[       OK ] StreamTest.StringIOTest (0 ms)
[----------] 1 test from StreamTest (0 ms total)

[----------] 1 test from FileBackedStreamTest
[ RUN      ] FileBackedStreamTest.FileBackedObjectIOTest
[       OK ] FileBackedStreamTest.FileBackedObjectIOTest (3 ms)
[----------] 1 test from FileBackedStreamTest (3 ms total)

[----------] 4 tests from ZipTest
[ RUN      ] ZipTest.CreateMember
zip_test.cc:46: Failure
Value of: file.get()
  Actual: false
Expected: true
Unable to create zip file
[  FAILED  ] ZipTest.CreateMember (2 ms)
[ RUN      ] ZipTest.OpenMemberByURN
zip_test.cc:46: Failure
Value of: file.get()
  Actual: false
Expected: true
Unable to create zip file
[  FAILED  ] ZipTest.OpenMemberByURN (3 ms)
[ RUN      ] ZipTest.ConcatenatedVolumes
zip_test.cc:46: Failure
Value of: file.get()
  Actual: false
Expected: true
Unable to create zip file
[  FAILED  ] ZipTest.ConcatenatedVolumes (3 ms)
[ RUN      ] ZipTest.testStreamedSegment
zip_test.cc:46: Failure
Value of: file.get()
  Actual: false
Expected: true
Unable to create zip file
[  FAILED  ] ZipTest.testStreamedSegment (3 ms)
[----------] 4 tests from ZipTest (11 ms total)

[----------] 2 tests from AFF4ImageTest
[ RUN      ] AFF4ImageTest.OpenImageByURN
[2019-01-23 17:29:18.562] [aff4] [error] Unable to open backing URN /tmp/aff4_test.zip
[2019-01-23 17:29:18.563] [aff4] [error] Unable to open backing URN /tmp/aff4_test.zip
[2019-01-23 17:29:18.563] [aff4] [error] Unable to open backing URN /tmp/aff4_test.zip
terminate called after throwing an instance of 'std::future_error'
  what():  std::future_error: No associated state
Aborted (core dumped)

Makefile:496: ../../aff4/.deps/linpmem-compatibility.Po: No such file or directory

I am trying to build pmem static libraries on Ubuntu17.10 again, but also occurs error.

Making install in aff4
make[1]: Entering directory '/home/meimei/c-aff4/aff4'
make[2]: Entering directory '/home/meimei/c-aff4/aff4'
/bin/mkdir -p '/home/meimei/static/lib'
/bin/mkdir -p '/home/meimei/static/include/aff4/'
/bin/bash ../libtool --mode=install /usr/bin/install -c libaff4.la '/home/meimei/static/lib'
/usr/bin/install -c -m 644 aff4_base.h aff4_image.h aff4_io.h aff4_simple.h config.h lexicon.h lexicon.inc rdf.h aff4_errors.h aff4_init.h aff4_registry.h aff4_utils.h data_store.h libaff4.h zip.h aff4_map.h tclap_parsers.h aff4_directory.h aff4_imager_utils.h aff4_file.h aff4_symstream.h libaff4-c.h threadpool.h '/home/meimei/static/include/aff4/'
libtool: install: /usr/bin/install -c .libs/libaff4.lai /home/meimei/static/lib/libaff4.la
libtool: install: /usr/bin/install -c .libs/libaff4.a /home/meimei/static/lib/libaff4.a
libtool: install: chmod 644 /home/meimei/static/lib/libaff4.a
libtool: install: ranlib /home/meimei/static/lib/libaff4.a
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/sbin" ldconfig -n /home/meimei/static/lib

Libraries have been installed in:
/home/meimei/static/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:

• add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
  during execution
• add LIBDIR to the 'LD_RUN_PATH' environment variable
  during linking
• use the '-Wl,-rpath -Wl,LIBDIR' linker flag
• have your system administrator add LIBDIR to '/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

/bin/mkdir -p '/home/meimei/static/bin'
/bin/bash ../libtool --mode=install /usr/bin/install -c aff4imager '/home/meimei/static/bin'
libtool: install: /usr/bin/install -c aff4imager /home/meimei/static/bin/aff4imager
make[2]: Leaving directory '/home/meimei/c-aff4/aff4'
make[1]: Leaving directory '/home/meimei/c-aff4/aff4'
Making install in tools/pmem
make[1]: Entering directory '/home/meimei/c-aff4/tools/pmem'
Makefile:496: ../../aff4/.deps/linpmem-compatibility.Po: No such file or directory
make[1]: *** No rule to make target '../../aff4/.deps/linpmem-compatibility.Po'. Stop.
make[1]: Leaving directory '/home/meimei/c-aff4/tools/pmem'
Makefile:465: recipe for target 'install-recursive' failed
make: *** [install-recursive] Error 1

../../aff4/.deps has files:
total 560
-rw-r--r-- 1 root root 29062 Mar 31 23:01 aff4_directory.Plo
-rw-r--r-- 1 root root 28275 Mar 31 23:01 aff4_file.Plo
-rw-r--r-- 1 root root 29286 Mar 31 23:00 aff4_image.Plo
-rw-r--r-- 1 root root 32628 Mar 31 23:01 aff4imager.Po
-rw-r--r-- 1 root root 34645 Mar 31 23:01 aff4_imager_utils.Plo
-rw-r--r-- 1 root root 83 Mar 31 22:59 aff4_init.Plo
-rw-r--r-- 1 root root 28914 Mar 31 23:01 aff4_map.Plo
-rw-r--r-- 1 root root 28113 Mar 31 23:01 aff4_symstream.Plo
-rw-r--r-- 1 root root 2758 Mar 31 23:01 compatibility.Po
-rw-r--r-- 1 root root 28973 Mar 31 23:00 data_store.Plo
-rw-r--r-- 1 root root 26788 Mar 31 23:00 lexicon.Plo
-rw-r--r-- 1 root root 28961 Mar 31 23:01 libaff4-c.Plo
-rw-r--r-- 1 root root 30226 Mar 31 23:00 libaff4.Plo
-rw-r--r-- 1 root root 27211 Mar 31 23:00 rdf.Plo
-rw-r--r-- 1 root root 19377 Mar 31 23:01 tclap_parsers.Plo
-rw-r--r-- 1 root root 30135 Mar 31 23:00 zip.Plo

Thanks a lot.

C API uses errno

The current C API returns -1 on failure and sets errno with a specific error value. This behavior should be avoided outside of system calls [1].

I suggest that the C API instead return an AFF4Status value, but am interested in your thoughts before I submit a PR.

[1] https://github.com/isocpp/CppCoreGuidelines/blob/master/CppCoreGuidelines.md#Re-no-throw

Be able to stream the image to stdout

Zip64 supports streaming to stdout and this should be implemented.

AFF4_read() unnecessarily copies data

AFF4_read copies data from a std::string to the caller-supplied buffer. It would be more efficient if instead the data were read into the caller-supplied buffer directly, to avoid copying.

output not to stdout? driver not loading

in KAPE, we have a module for winpmem, but it is running and exiting immediately. i do not see any info being dumped to the console tho. when i run it directly, i get this output:

this is with 3.2, but 3.3 does the same thing.

any ideas? is winpmem writing to stdout?

linpmem-v3.3-rc and GLIBC dependency issues

In a previously closed thread found here:
#25

It was identified as a commented out line that solved the dependency issue. There is a "linpmem.gz" attachment dated April 26, 2018 that no longer has the GLIBC dependency issue.

When testing the latest version, v3.3-rc1 on a RedHat Enterprise Linux 7.7 system, the following errors are being dispayed:

linpmem: /lib64/libc.so.6: version GLIBC_2.27' not found (required by linpmem) linpmem: /lib64/libc.so.6: version GLIBC_2.25' not found (required by linpmem)

it would appear that the previously solved issue has returned in the latest release.

I have downloaded that GZ file from the closed issue posting and it DOES work on my RHEL system.
Please advise

-Lou

Build failing due to the presence of C++14 code (std::make_unique)

I'm trying to build the current master (aa400e8) on OSX 10.13.2, but having no luck. It seems as though autoconf is configured for C++11, but there is C++14 code in the codebase.

Here's a gist of my autogen, configure, and make: https://gist.github.com/justfalter/e4393ad33afc44c2cf8fd28b0d2b628a

The error that it's bombing out on is the use of std::make_unique (here), which according to http://en.cppreference.com/w/cpp/memory/unique_ptr/make_unique was introduced with C++14.

      pool(std::make_unique<ThreadPool>(options.threadpool_size)),
           ~~~~~^
data_store.cc:124:29: error: 'ThreadPool' does not refer to a value
      pool(std::make_unique<ThreadPool>(options.threadpool_size)),
                            ^```

autogen errors

I get the following output when attempting to run autogen.sh on macOS.

autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4 ${ACLOCAL_FLAGS}
autoreconf: configure.ac: tracing
autoreconf: running: glibtoolize --copy --force
glibtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'config'.
glibtoolize: copying file 'config/ltmain.sh'
glibtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
glibtoolize: copying file 'm4/libtool.m4'
glibtoolize: copying file 'm4/ltoptions.m4'
glibtoolize: copying file 'm4/ltsugar.m4'
glibtoolize: copying file 'm4/ltversion.m4'
glibtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/local/Cellar/autoconf/2.69/bin/autoconf --force
configure.ac:15: error: possibly undefined macro: AC_SUBST
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
configure.ac:60: error: possibly undefined macro: AC_MSG_ERROR
configure.ac:121: error: possibly undefined macro: AC_DEFINE
autoreconf: /usr/local/Cellar/autoconf/2.69/bin/autoconf failed with exit status: 1

yaml-cpp always required, but build fails if using --with-yaml

If I configure the latest c-aff4 with --with-yaml (I have yaml-cpp 0.6.2 installed), the build fails with:

In file included from data_store.cc:32:
In file included from /opt/local/include/yaml-cpp/yaml.h:13:
In file included from /opt/local/include/yaml-cpp/stlemitter.h:12:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/set:398:40: error: no template named 'less'; did you mean '::std::less'?
template <class _Key, class _Compare = less<_Key>,
                                       ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__functional_base:57:29: note: '::std::less' declared here
struct _LIBCPP_TEMPLATE_VIS less : binary_function<_Tp, _Tp, bool>
                            ^

(followed by lots of other errors)

If I do not use --with-yaml, then the build succeeds, but yaml-cpp is still required at build time, and osxpmem does end up linked with the yaml-cpp library.

Finish removing remove libpcre++ dependency

I was about to update the docker image to remove the libpcre++ dependency building (in light of #56), when I noticed that the dependency still exists in linux_pmem.cc.

AFAICT, with the changes that had been made in #56, master will probably fail to build linux_pmem.

osxpmem, Loading/Unloading kext

I posted this to the working group, but may be better off here with more administrative issues over there. Also tl;dr material... sorry.

Here's a rundown of my testing of the latest AFF4/osxpmem binary for MacOS/APFS so far (also, this is for mem capture, not logical imaging):

Macbook Pro
MacOS High Sierra 10.13.4
APFS (HFS+ on the MACTOOLS volume and ExFAT on the EVIDENCE volume)

I put the osxpmem.zip in an HFS+ formatted volume on an external HDD labeled MACTOOLS, along with the test directory scheme I used for the linpmem tests. This HDD also has a volume labeled EVIDENCE (ExFAT) and I put a /test_mac directory there to catch output. I also set the OS to not ignore ownership on the MACTOOLS volume. I first unzipped the archive:

$ unzip osxpmem.zip

...then checked ownership:

$ ls -Fla
total 2929
drwxr-xr-x 9 hharness staff 306 Apr 20 09:01 ./
drwxr-xr-x 11 hharness staff 442 Apr 20 08:30 ../
-rw-r--r--@ 1 hharness staff 6148 Apr 20 09:01 .DS_Store
drwxr-xr-- 6 hharness staff 204 Feb 9 00:03 osxpmem.app/
-rw-r--r--@ 1 hharness staff 1562910 Apr 19 17:11 osxpmem.zip
-rw-r--r--@ 1 hharness staff 20 Apr 20 08:28 singlefile.txt
drwxr-xr-x 6 hharness staff 204 Apr 15 17:34 testdir/

I changed ownership as follows:

$ sudo chown -R root:wheel osxpmem.app

I ran it using:

$ sudo ./osxpmem.app/osxpmem -o /Volumes/EVIDENCE/test_mac/macram.aff4

This gave the following results:

2018-04-20 09:17:18 E Can not open file /dev/pmem: No such file or directory
/Volumes/MACTOOLS/MACTEST/./osxpmem.app/MacPmem.kext failed to load -
(libkern/kext) system policy prevents loading; check the system/kernel logs for errors or try kextutil(8).
2018-04-20 09:17:19 E Unable to load driver at /Volumes/MACTOOLS/MACTEST/./osxpmem.app/MacPmem.kext
2018-04-20 09:17:19 E Imaging failed with error: -8

I went ahead and tried to load the kext manually and got this:

$ sudo kextutil osxpmem.app/MacPmem.kext

Kext rejected due to improper filesystem permissions: <OSKext 0x7fa0a9466e40 [0x7fff87d33af0]> { URL =
"file:///Library/StagedExtensions/Volumes/MACTOOLS/MACTEST/osxpmem.app/MacPmem.kext/", ID = "com.google.MacPmem" }
Diagnostics for osxpmem.app/MacPmem.kext:
Authentication Failures:
File owner/permissions are incorrect (must be root:wheel, nonwritable by group/other):
/Library/StagedExtensions/Volumes/MACTOOLS/MACTEST/osxpmem.app/MacPmem.kext
Contents
_CodeSignature
CodeResources
MacOS
MacPmem
Info.plist

I did add the security exception for the kext signed by Adam Sindelar. See here: https://developer.apple.com/library/content/technotes/tn2459/_index.html

Checking ownership again:

$ ls -Fla
total 2929
drwxr-xr-x 9 1000 1000 306 Apr 20 09:14 ./
drwxr-xr-x 12 1000 1000 476 Apr 20 10:00 ../
-rw-r--r--@ 1 hharness staff 6148 Apr 20 09:14 .DS_Store
drwxr-xr-- 6 root wheel 204 Feb 9 03:03 osxpmem.app/
-rw-r--r--@ 1 1000 1000 1562910 Apr 19 17:11 osxpmem.zip
-rw-r--r--@ 1 1000 1000 20 Apr 20 08:28 singlefile.txt
drwxr-xr-x 6 1000 1000 204 Apr 15 17:34 testdir/

I escalated with sudo su, navigated through osxpmem.app, and confirmed the owner:group and nowrite for group/other was set properly. So, ownership shouldn't be the issue and neither should system policy prevention based on user approval. The message I received by attempting to run osxpmem allowing it to autoload the kext gave an illegal path of /Volumes/MACTOOLS/MACTEST/./osxpmem.app/MacPmem.kext (see above). I don't know where this path is coming from, but might be a clue if that's what it's actually trying to follow. It doesn't explain the error message when trying to load the kext manually with kextutil.

Custom Driver Extraction Path

Currently the winpmem driver extracts to temp which is causing issues with av. Would it be possible to get a command line flag implemented to allow a custom path for driver extraction.

how to acquire disk images like(/dev/sda) on windows,or can it work?

E:>aff4imager.exe -dd -o E:\output.aff4 -i /dev/sde --format raw --volume_format raw
2020-03-27 13:28:35 I This is The WinPmem memory imager. version 3.3rc2
2020-03-27 13:28:35 I Output volume is not an AFF4 file. Cannot capture additional streams. Choose an AFF4 volume to capture additional streams.

osxpmem "Cannot open file /dev/pmem: No such file or directory" message error is confusing

The error message "E Cannot open file /dev/pmem: No such file or directory" can be confusing, especially because it is the only feedback given to the user in non-verbose or non-debug mode.

In fact it looks like something bad happened during the dump process whereas it is just telling the user /dev/pmem does not exist yet and osxpmem needs to load its driver to use it.

I think the message should be more explicit.

Dev Docs: Where to start?

Hola,

I'd potentially like to use aff4 in a new project, but I'm not sure where to start.

• Where can I read the docs?
• Who's in charge?
• What policies apply? (I note my team member has made significant contributions and had to change them to conform with the notoriously C++-hostile Google C++ "Style" Guidelines. But then... it seems... the API is C++-based? So there are ABI breakage issues?)
• What works?
• What needs to be fixed?

The possible universe where AFF4 becomes a common alternative to E01/L01 is the one I want to live in and I'm willing to invest in it becoming a reality, but I am a conservative investor.

cheers,

Jon

Missing configuration check for spdlog

spdlog is required to compile, but there's no autoconf check included.