volatilityfoundation / dwarf2json Goto Github PK

View Code? Open in Web Editor NEW

93.0 93.0 26.0 62 KB

convert ELF/DWARF symbol and type information into vol3's intermediate JSON

License: Other

Go 100.00%

dwarf2json's People

Contributors

Stargazers

Watchers

dwarf2json's Issues

Provide feedback when only symbols or only types are produced

There may be an expectation that just providing an elf file, or just providing the system.map will be enough to generate a complete JSON file. It would be awesome to be able to let the user know if the JSON file they've generated won't necessarily work properly (ie, lacking symbols, lacking types, lacking banner/constant_data).

Mac 10.9 symbols don't produce version constant_data fields

Generating JSON files from all the images, 10.4 through to 10.14, all the 10.9 images fail (both the base mach_kernel and also the DEBUG_kernel/mach_kernel). Happy to run tests, but all the files seem to be in the same places as the others, so not sure what's going on there...

struct name lost?

when struct defined in this style:
tyedef struct{
...
}aaa;
aaa bbb;
The output json will not contain the name "aaa", just show unamed_xxxx. So the json can output the real struct name directly?
output such as:
"bbb": {
"type": {
"kind": "struct",
"name": "aaa"
},
"offset": 7
}

Bitfield with negative bit_position value

Hi,
generating the json profile for vmlinux-5.19.0-40-generic Ubuntu 22.04 stock kernel I have discovered that the tool produce a negative bit_position value for bitfields (example for field active_mode in struct Scsi_Host.
I think the problem come from this line:

dwarf2json/main.go

Line 290 in c306d11

    
           vtypeField.FieldType["bit_position"] = maxPos - (field.BitOffset + (field.BitSize - 1))

Please can you fix it?

Process Crash Due to Insufficient Memory

This issue reports that the process crashes when the system's available RAM falls below a certain threshold.
While creating a large swap file (e.g., 8GB) can mitigate the issue, it's considered poor practice due to potential performance drawbacks.

Missing fields in complex structs with nested anonymous unions/structs

Hi,
the tool seems not able to parse correctly structures with a lot of anonymous unions/structures embedded inside a struct.
For example the tool for struct page, which contains a lot of anonymous unions/structs nested, produce a JSON with very few fields..
Can you fix it?

Mac < 10.7 kernels are store in dual-architecture files which aren't read correctly

Hiya,

As promised, here's a bug about reading Mach-0 dual architecture files. dSYM files before 10.8 come with both 64bit and 32bit files. I dunno if it's worth generating symbols for that long ago, or whether it's just worth doing one or the other architecture, but either way currently dwarf2json says it's not a valid Mach-O, ELF or DWARF file...

kernel_debug_kit_10.7.5_11g56.dmg/System/DEBUG_Kernel/mach_kernel.dSYM/Contents/Resources/DWARF/mach_kernel:   Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit x86_64 dSYM companion file] [i386]
kernel_debug_kit_10.8.1_12b19.dmg/System/DEBUG_Kernel/mach_kernel.dSYM/Contents/Resources/DWARF/mach_kernel:   Mach-O 64-bit x86_64 dSYM companion file

runtime: goroutine stack exceeds 1000000000-byte limit

gara@z840:~/Documents/dwarf2json$ du -h /home/gara/Downloads/red8/usr/lib/debug/lib/modules/4.18.0-240.15.1.el8_3.x86_64/vmlinux
780M    /home/gara/Downloads/red8/usr/lib/debug/lib/modules/4.18.0-240.15.1.el8_3.x86_64/vmlinux


gara@z840:~/Documents/dwarf2json$ ./dwarf2json linux --elf /home/gara/Downloads/red8/usr/lib/debug/lib/modules/4.18.0-240.15.1.el8_3.x86_64/vmlinux > r8.json
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc0521803a0 stack=[0xc052180000, 0xc072180000]
fatal error: stack overflow

runtime stack:
runtime.throw(0x56280f, 0xe)
        /snap/go/7013/src/runtime/panic.go:1116 +0x72
runtime.newstack()
        /snap/go/7013/src/runtime/stack.go:1067 +0x78d
runtime.morestack()
        /snap/go/7013/src/runtime/asm_amd64.s:449 +0x8f

goroutine 1 [running]:
runtime.mallocgc(0x9, 0x0, 0x40e100, 0xc00d714800)
        /snap/go/7013/src/runtime/malloc.go:903 +0xa8b fp=0xc0521803b0 sp=0xc0521803a8 pc=0x40e86b
runtime.rawstring(0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
        /snap/go/7013/src/runtime/string.go:263 +0x4f fp=0xc0521803e0 sp=0xc0521803b0 pc=0x44f1ef
runtime.rawstringtmp(0x0, 0x9, 0x44ebb4, 0x7, 0xc0293f0210, 0x7, 0xc0293f0210)
        /snap/go/7013/src/runtime/string.go:131 +0x74 fp=0xc052180420 sp=0xc0521803e0 pc=0x44ebb4
runtime.concatstrings(0x0, 0xc052180500, 0x3, 0x3, 0xc0293f0210, 0xc0293f0215)
        /snap/go/7013/src/runtime/string.go:50 +0xc5 fp=0xc0521804b8 sp=0xc052180420 pc=0x44e605
runtime.concatstring3(0x0, 0x560823, 0x6, 0x5600a8, 0x1, 0xc018eb017c, 0x2, 0xc0293f0210, 0x7)
        /snap/go/7013/src/runtime/string.go:63 +0x47 fp=0xc0521804f8 sp=0xc0521804b8 pc=0x44e907
debug/dwarf.(*StructType).String(0xc0001b8ea0, 0x56056e, 0x5)
        /snap/go/7013/src/debug/dwarf/type.go:164 +0x71 fp=0xc052180550 sp=0xc0521804f8 pc=0x4cb1d1
debug/dwarf.(*StructType).Defn(0xc018fca840, 0xc052180640, 0x44e887)
        /snap/go/7013/src/debug/dwarf/type.go:183 +0x106 fp=0xc0521805f8 sp=0xc052180550 pc=0x4cb346
debug/dwarf.(*StructType).String(0xc018fca840, 0x560823, 0x6)
        /snap/go/7013/src/debug/dwarf/type.go:166 +0x98 fp=0xc052180650 sp=0xc0521805f8 pc=0x4cb1f8
debug/dwarf.(*StructType).Defn(0xc018fca7e0, 0xc052180740, 0x44e887)
        /snap/go/7013/src/debug/dwarf/type.go:183 +0x106 fp=0xc0521806f8 sp=0xc052180650 pc=0x4cb346
debug/dwarf.(*StructType).String(0xc018fca7e0, 0xc0293ead80, 0xbd)
        /snap/go/7013/src/debug/dwarf/type.go:166 +0x98 fp=0xc052180750 sp=0xc0521806f8 pc=0x4cb1f8
debug/dwarf.(*StructType).Defn(0xc00012f140, 0xc052180840, 0x44e887)

Certain mac symbols still have type null

Having generated all the JSON from the KDKs, it seems there are five specific symbols that always come up with a "type": null, which then breaks jsonschema validation. Some files only contain the kOSBoolean values, but regardless that's enough to break the validation.

The five are

func
kOSBooleanFalse
kOSBooleanTrue
method
trap

The files otherwise pass validation, so this should be the last hurdle. 5:)

How to get symbol of a NAS system

golang env:

GOVERSION='go1.21.1'
GO111MODULE='on'
GOARCH='amd64'
GOOS='linux'

readelf for nas -> ash:

ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x41fef4
  Start of program headers:          64 (bytes into file)
  Start of section headers:          1023192 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         26
  Section header string table index: 25

run

root@nas:/volume1/share# ./dwarf2json 
./dwarf2json: /lib/libc.so.6: version `GLIBC_2.34' not found (required by ./dwarf2json)
./dwarf2json: /lib/libc.so.6: version `GLIBC_2.32' not found (required by ./dwarf2json)

system

root@nas:/volume1/share# cat /proc/version 
Linux version 4.4.59+ (root@build11) (gcc version 4.9.3 20150311 (prerelease) (crosstool-NG 1.20.0) ) #24922 SMP PREEMPT Fri May 10 02:59:42 CST 2019
root@nas:/volume1/share# uname -a
Linux nas 4.4.59+ #24922 SMP PREEMPT Fri May 10 02:59:42 CST 2019 x86_64 GNU/Linux synology_apollolake_918+

I want to get the symbols of the nas system to analyze the memory data dumped in the nas. I tried the above operations, but I am stuck at this step. I don’t know what I should do next.

openSUSE Tumbleweed: decoding dwarf section underflow failure

An attempt to create a full profile for the Linux 5.8.0 kernel on openSUSE Tumbleweed fails with:

$ dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.8.0-1-default.debug > 5.8.0-1-default.json
Failed linux processing: could not get DWARF from /usr/lib/debug/boot/vmlinux-5.8.0-1-default.debug: decoding dwarf section info at offset 0x2705: underflow

OTOH, a pure symbol mapping succeeds:

$ dwarf2json linux --system-map /boot/System.map-5.8.0-1-default > 5.8.0-1-default-syms.json

If you want to reproduce the issue, fetch the kernel debug package from here. The filename might change in case of new releases (as being expected). You should be able to convert this with alien to a regular tarball, or use rpm2cpio file | cpio -idmv.

dwarf2json was built directly from the git tree here.

Being a rolling release, we tend to use a pretty current toolchain. If you need more details, let me know.

Unable to validate the plugin requirements

Describe the bug

Similar to volatilityfoundation/volatility3#634, I am getting the following error message (with logs for context):

$ python3 vol.py -f /home/aaron/Downloads/HTB/forensics_poof/mem.dmp -vvvv windows.pslist.PsList
Volatility 3 Framework 2.4.0
INFO     volatility3.cli: Volatility plugins path: ['/home/aaron/Documents/CYBER/volatility3/volatility3/plugins', '/home/aaron/Documents/CYBER/volatility3/volatility3/framework/plugins']
INFO     volatility3.cli: Volatility symbols path: ['/home/aaron/Documents/CYBER/volatility3/volatility3/symbols', '/home/aaron/Documents/CYBER/volatility3/volatility3/framework/symbols']
INFO     volatility3.framework.automagic: Detected a windows category plugin
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.kernel
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.PsList
INFO     volatility3.framework.automagic: Running automagic: SymbolCacheMagic
INFO     volatility3.framework.automagic: Running automagic: LayerStacker
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 8  volatility3.framework.automagic.stacker: Stacked Elf64Layer using Elf64Stacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsIntelStacker
DEBUG    volatility3.framework.automagic.windows: Detecting Self-referential pointer for recent windows
DEBUG    volatility3.framework.automagic.windows: Older windows fixed location self-referential pointers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: TypeError - Layer is not the required Architecture: Elf64Layer
Level 9  volatility3.framework.configuration.requirements: TypeError - Layer is not the required Architecture: FileLayer
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['Elf64Layer', 'FileLayer']
INFO     volatility3.framework.automagic: Running automagic: WinSwapLayers
INFO     volatility3.framework.automagic: Running automagic: KernelPDBScanner
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
INFO     volatility3.framework.automagic.pdbscan: No suitable kernels found during pdbscan
INFO     volatility3.framework.automagic: Running automagic: SymbolFinder
INFO     volatility3.framework.automagic: Running automagic: KernelModule
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.PsList.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.PsList.kernel.symbol_table_name

Unsatisfied requirement plugins.PsList.kernel.layer_name:
Unsatisfied requirement plugins.PsList.kernel.symbol_table_name:

A translation layer requirement was not fulfilled.  Please verify that:
A file was provided to create this layer (by -f, --single-location or by config)
The file exists and is readable
The file is a valid memory image and was acquired cleanly

A symbol table requirement was not fulfilled.  Please verify that:
The associated translation layer requirement was fulfilled
You have the correct symbol file for the requirement
The symbol file is under the correct directory or zip file
The symbol file is named appropriately or contains the correct banner

Unable to validate the plugin requirements: ['plugins.PsList.kernel.layer_name', 'plugins.PsList.kernel.symbol_table_name']

Context
Volatility Version: Latest
Operating System: ZorinOS 16 Pro
Python Version: 3.9.12
Suspected Operating System: Ubuntu 4.15.0-184
Command: python3 vol.py -f /home/aaron/Downloads/HTB/forensics_poof/mem.dmp -vvvv windows.pslist.PsList

To Reproduce

I tried generating a JSON file with ./dwarf2json linux --system-map ./System.map-4.15.0-184-generic > System.map-4.15.0-184-generic.json and copied System.map-4.15.0-184-generic.json, System.map-4.15.0-184-generic, and module.dwarf to both /volatility3/volatility3/symbols and /volatility3/volatility3/framework/symbols/linux to cover all my bases. Doing so yielded the error message above.

Both of these files had a .txt extension added so they could be uploaded to GitHub, on my system they do not have the extension.

System.map-4.15.0-184-generic.txt
module.dwarf.txt

Expected behavior

I expect the symbols to be detected and to allow me to perform an analysis of the memory dump.

Additional information

I know the mem.dmp file is formatted correctly and I know the System Map and Dwarf files are good, they were all provided by Hack The Box and others have been able to solve this challenge. I'm not sure if there is some incompatibility with Zorin or if I'm just making some silly mistake, but any support that you could provide would be greatly appreciated :)

Build issue w/ go

Generate symbols for Solus linux

Hi,

I'm trying to generate a symbol table for Solus Linux and there is this package:
linux-current-dbginfo-5.15.61-217-1-x86_64.eopkg - Debug symbols for linux-current.
but not actual debug kernel as discussed in #37 .

Is the debug kernel mandatory? Can I use the current information to built the table ?
PS: I'm using https://github.com/microsoft/avml to create the memory dump.

Cheers,
PY

Modern linux types can sometimes just be levels of indirection

Re: volatilityfoundation/volatility3#151

No user types being generated when using System map + dwarf

Dwarf was generated using vol2's tools/linux.

Using Ubuntu 18.

If I use vmlinux it works but shouldn't just a dwarf and map be enough?

Command issued"
./dwarf2json linux --system-map System.map-4.15.0-106-generic module.dwarf > output.json

I have included an upload to the files:
dwarf_and_map.zip

unknown ELF version 'EV_NONE' in record at byte 0x0

Hi Dear,

I have cloned the tool
then, I did go build
then, sent the command .\dwarf2json.exe linux --elf C:/Users/User/Dumps/mem.elf > output.json

and i received this error:

Failed linux processing: could not open C:/Users/User/Dumps/mem.elf: unknown ELF version 'EV_NONE' in record at byte 0x0

What can be the problem?

d.AddSection undefined (type *dwarf.Data has no field or method AddSection)

Trying to build the latest master branch results in the error

./dwarf.go:108:15: d.AddSection undefined (type *dwarf.Data has no field or method AddSection)

Generate symbols arch

Hi I'm having problems to create the symbols for the Raspberry Pi5 Linux version 6.1.58-v8+ (Kali linux 2023.3) any advice?

bitfields have wrong bit position

struct printk_log {
  u64 ts_nsec;    /* timestamp in nanoseconds */
  u16 len;    /* length of entire record */
  u16 text_len;   /* length of text buffer */
  u16 dict_len;   /* length of dictionary buffer */
  u8 facility;    /* syslog facility */
  u8 flags:5;   /* internal record flags */
  u8 level:3;   /* syslog level */
#ifdef CONFIG_PRINTK_CALLER
  u32 caller_id;            /* thread id or processor id */
#endif
}

Debugging the Linux kernel I got:

>>> p msg->flags
$14 = 6 '\006'

>>> p msg->level
$15 = 5 '\005'

>>> p *(unsigned char*)&msg->flags
$16 = 166 '\246'

Debugging Volatility we can see we got wrong values:

> msg.flags
20
> msg.level
6

From the ISF we can see that the bit_position of flags and level are 3 and 0 respectively.

    "printk_log": {
        ...
        "flags": {
          "type": {
            "bit_length": 5,
            "bit_position": 3,
            "kind": "bitfield",
            "type": {
              "kind": "base",
              "name": "unsigned char"
            }
          },
          "offset": 15
        },
        "level": {
          "type": {
            "bit_length": 3,
            "bit_position": 0,
            "kind": "bitfield",
            "type": {
              "kind": "base",
              "name": "unsigned char"
            }
          },
          "offset": 15
        },
    ...
    }

So, due to the wrong ISF bit_position values, Volatility seems to be doing the following:

>>> 166 & 0b111
6
>>> 166>>3 & 0b11111
20

OS Debug Symbols: Linux / Ubuntu 18.10
Kernel: 4.18.0-10-generic
Kernel debug symbols package: linux-image-unsigned-4.18.0-10-generic-dbgsym

linux-image-4.18.0-10-generic.zip

go.mod: checksum mismatch

I encountered this error today when trying to install dwarf2json:

dwarf2json# go build
verifying github.com/spf13/[email protected]/go.mod: checksum mismatch
downloaded: h1:G7mAYYxgmS0lVkHyy2hEOLQCFB0DlQFTMLWggykrydY=
go.sum: h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

dwarf2json feature request for vol2 profiles

I would like to submit a feature request for when you have access only to the vol2 profile which includes the systemmap and the module.dwarf. Is that enough to create the profile for Vol3? That would be very helpful in converting vol2 profiles to vol3 if it is possible.

Process Killed and empty output file

I am trying to make an isf json file using this command:

./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-1070-azure --system-map /boot/System.map-5.4.0-1070-azure > ubuntu18.04.6.json

... and the process always turns a "Killed" and the output file will be empty.

I tried

./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-1070-azure --system-map /boot/System.map-5.4.0-1070-azure >  | xz -c ubuntu18.04.6.json.xz

as well, and it did not fail, but the output .json.xz file was warned with "invalid isf file" when I checked them with isfinfo.IsfInfo of Volatility3.

Is there anything wrong with my command?

`fs_struct` type is not converted correctly

For Ubuntu 23.10 the fs_struct type is not converted correctly. No clue if this is the only type that is affected, or if there is a bigger issue. Using commit 15baf31210af272dc2b79ba79a6e58abc5d0336b

You can download the kernel+map here.

$ llvm-dwarfdump -r 1 ../kernels/ubuntu-23.10-live-server-amd64-0052522788-vmlinux-6.5.0-14-generic-2e44e-2e44e.elf | head | grep Compile
0x00000000: Compile Unit: length = 0x0002592c, format = DWARF32, version = 0x0005, unit_type = DW_UT_compile, abbr_offset = 0x0000, addr_size = 0x08 (next unit at 0x00025930)
$ gdb -n -q --batch -ex 'ptype /r init_task.fs' -ex "quit" ../kernels/ubuntu-23.10-live-server-amd64-0052522788-vmlinux-6.5.0-14-generic-2e44e-2e44e.elf
type = struct fs_struct {
    int users;
    spinlock_t lock;
    seqcount_spinlock_t seq;
    int umask;
    int in_exec;
    struct path root;
    struct path pwd;
} *
$ pahole -F dwarf ../kernels/ubuntu-23.10-live-server-amd64-0052522788-vmlinux-6.5.0-14-generic-2e44e-2e44e.elf | rg '^struct fs_struct \{' -A 15
struct fs_struct {
        int                        users;                /*     0     4 */
        spinlock_t                 lock;                 /*     4     4 */
        seqcount_spinlock_t        seq;                  /*     8     4 */
        int                        umask;                /*    12     4 */
        int                        in_exec;              /*    16     4 */

        /* XXX 4 bytes hole, try to pack */

        struct path                root;                 /*    24    16 */
        struct path                pwd;                  /*    40    16 */

        /* size: 56, cachelines: 1, members: 7 */
        /* sum members: 52, holes: 1, sum holes: 4 */
        /* last cacheline: 56 bytes */
};
# ./volshell.py -f /io/dumps/ubuntu-23.10-live-server-amd64-0052522788_19.raw -l --script <(echo -e "dt('fs_struct')\nquit()")
Volshell (Volatility 3 Framework) 2.5.2
Readline imported successfully  Stacking attempts finished
Running code from file:///dev/fd/63

symbol_table_name1!fs_struct (0 bytes)
 0x0 :   _unused     symbol_table_name1!array
$ cat ubuntu-23.10-live-server-amd64-0052522788-vmlinux-6.5.0-14-generic-2e44e-2e44e.isf.json| rg '"fs_struct": \{' -A 16
    "fs_struct": {
      "size": 0,
      "fields": {
        "_unused": {
          "type": {
            "count": 0,
            "kind": "array",
            "subtype": {
              "kind": "base",
              "name": "u8"
            }
          },
          "offset": 0
        }
      },
      "kind": "struct"
    },

How to query ASLR slide of load/stack address in arange ?

Thank you for making this tool open, it's helpful for symbolizing iOS crash logs.

While I have not found how to query ASLR slide of a given load/stack address yet.
Because of the ASLR slides, I cannot get the correct symbolicated source code.

I have tried editing some source code of the tool but it did not work, I still can not get the ASLR slides,
neither the .debug_arange which were told that contains ASLR slides for load/stack address.

Please add such a function or tell me how to. Thank you very much~

References:
https://eli.thegreenplace.net/2011/12/26/the-contents-of-dwarf-sections
https://bellis1000.medium.com/aslr-the-ios-kernel-how-virtual-address-spaces-are-randomised-d76d14dc7ebb
https://en.wikipedia.org/wiki/Address_space_layout_randomization#iOS_(iPhone,_iPod_touch,_iPad)

'go build' command fails

From following your README.md to build dwarf2json it says run go build to build it but after installing go (go version go1.6.2 linux/amd64) I get the following error:
main.go:28:2: cannot find package "github.com/spf13/pflag" in any of: /usr/lib/go-1.6/src/github.com/spf13/pflag (from $GOROOT) ($GOPATH not set)

Can you please elaborate on your instruction on how to build dwarf2json?

Issue running "go build"

I am trying to build dwarf2json. When I run go build I get the following error message:

# /usr/lib/go/src/pflag
./dwarf.go:108:15: d.AddSection undefined (type *dwarf.Data has no field or method AddSection)

My go version is go 1.13.8.

Context
Volatility Version: 2
Operating System: Ubuntu 20
Python Version: 2.7.18
Command: go build

Implement Progress Bar for Improved User Experience

Currently, the project lacks a progress bar, making it difficult for users to gauge the utility's progress, especially for long-running tasks. Implementing a progress bar would significantly enhance the user experience by providing:

Transparency: Users can easily visualize the progress of the operation.
Estimated Completion Time: A progress bar can offer an estimated time of completion, reducing user anxiety and allowing them to plan accordingly.

What is the appropriate kernel ELF to use with dwarf2json?

I'm trying to create a symbol table for my Debian distro. The example in the Readme doesn't work for Debian (since there is no /usr/lib/debug directory).

I looked up where to find the kernel ELF and landed on this StackOverflow question which gave two answers: /boot/vmlinuz-*, and a script to extract vmlinux.

The first case (/boot/vmlinuz-4.19.0-16-amd64 for me) points to an ELF (according to file), but dwarf2json doesn't like it:

$ ./dwarf2json linux --elf /boot/vmlinuz-4.19.0-16-amd64
Failed linux processing: could not open /boot/vmlinuz-4.19.0-16-amd64: bad magic number '[77 90 234 7]' in record at byte 0x0

The second case (vmlinux produced by the script) is also definitely an ELF (once again, according to file), but once again, dwarf2json rejected it:

$ ./extract-vmlinux.sh /boot/vmlinuz-4.19.0-16-amd64 > vmlinux
$ ./dwarf2json linux --elf vmlinux
Failed linux processing: could not get DWARF from vmlinux: decoding dwarf section info at offset 0x0: too short

For sake of interest, here's what file had to say for the two ELF cases above:

$ file /boot/vmlinuz-4.19.0-16-amd64
/boot/vmlinuz-4.19.0-16-amd64: Linux kernel x86 boot executable bzImage, version 4.19.0-16-amd64 ([email protected]) #1 SMP Debian 4.19.181-1 (2021-03-19), RO-rootFS, swap_dev 0x5, Normal VGA

$ file vmlinux
vmlinux: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=dbfab4e43b7d635bce4ac2cb0420e4dfab8049fd, stripped

The Volatility 3 documentation on this topic has exactly one sentence of wisdom to offer:

Once a kernel with debugging symbols/appropriate DWARF file has been located, dwarf2json will convert it into an appropriate JSON file.

Which doesn't really help.

Where should I get the appropriate kernel ELF for my distro so that I can generate the appropriate ISF for vol3?

user_types are lacking fields

The latest commit (72ea033) prevents strucutres fields being populated. master~1 works as expected.

dwarf2json linux --elf ./vmlinux # from http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-unsigned-4.15.0-108-generic-dbgsym_4.15.0-108.109_amd64.ddeb

Handle Rust bindings in the Linux kernel

Hi,

while investigating #57, I noticed the issue started appearing around the integration of Rust in the Linux kernel. With a bit more debugging, I was able to confirm that some bindings were being processed by dwarf2json in the same pool as C structs names :

$ llvm-dwarfdump vmlinux-6.5.0-14-generic --name fs_struct --show-children | grep 'DW_TAG_member' -A 2
0x063c0c3e:   DW_TAG_member
                DW_AT_name      ("_unused")
                DW_AT_type      (0x063ca2a5 "u8[0]")
--
0x063d404b:   DW_TAG_member
                DW_AT_name      ("_unused")
                DW_AT_type      (0x063d52f5 "u8[0]")

$ llvm-dwarfdump vmlinux-6.5.0-14-generic --debug-info=0x063c0c3e --show-parents
vmlinux-6.5.0-14-generic:       file format elf64-x86-64

.debug_info contents:

0x063b59a4: DW_TAG_compile_unit
              DW_AT_producer    ("clang LLVM (rustc version 1.68.2 (9eb3afe9e 2023-03-27) (built from a source tarball))")
              DW_AT_language    (DW_LANG_Rust)
              DW_AT_name        ("/build/linux-SXblTa/linux-6.5.0/rust/bindings/lib.rs/@/bindings.04c8d523-cgu.0")
              DW_AT_stmt_list   (0x00d8f3e1)
              DW_AT_comp_dir    ("/build/linux-SXblTa/linux-6.5.0/debian/build/build-generic")
              DW_AT_GNU_pubnames        (true)
              DW_AT_low_pc      (0xffffffff818060b0)
              DW_AT_high_pc     (0xffffffff81808cf0)

0x063be675:   DW_TAG_namespace
                DW_AT_name      ("bindings")

0x063be67a:     DW_TAG_namespace
                  DW_AT_name    ("bindings_raw")

0x063c0c37:       DW_TAG_structure_type
                    DW_AT_name  ("fs_struct")
                    DW_AT_byte_size     (0x00)
                    DW_AT_alignment     (1)

0x063c0c3e:         DW_TAG_member
                      DW_AT_name        ("_unused")
                      DW_AT_type        (0x063ca2a5 "u8[0]")
                      DW_AT_alignment   (1)
                      DW_AT_data_member_location        (0x00)

Should these bindings, or wider all rust content, be processed separately from the regular structures ? I don't think they should be discarded, but maybe stored under a different parent key in the ISF ?

panic: runtime error: invalid memory address or nil pointer dereference

Hey, given the following vmlinux :

$ file vmlinux-debug-fedora-kernel-6.0.7-301 
vmlinux-debug-fedora-kernel-6.0.7-301: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=6abfb4801ce2a8170450a819a4671d62430816ef, with debug_info, not stripped

I got the following runtime panic :

$ ./dwarf2json linux --elf ./vmlinux-debug-fedora-kernel-6.0.7-301 > fedora-kernel-6.0.7-301.json       
             
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x4cd262]

goroutine 1 [running]:
debug/dwarf.zeroArray(0xc050678010)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:744 +0x62
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01834d0e0, 0x453fcd3, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:579 +0xfb6
debug/dwarf.(*Data).readType.func3(0xc01c79a390, 0xc000000049, 0xc05061bb20)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01834d040, 0x453fdb4, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:644 +0x1f03
debug/dwarf.(*Data).readType.func3(0xc01c79a360, 0xc05061bac0, 0x0)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01834cd20, 0x453fcae, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:535 +0x518
debug/dwarf.(*Data).readType.func3(0xc01c79a270, 0xc05061b980, 0x0)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01834cbe0, 0x453fdb9, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:535 +0x518
debug/dwarf.(*Data).readType.func3(0xc01c79a1e0, 0xc00007a6c0, 0x453fddc)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01834cb40, 0x453fddc, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:406 +0x16a3
debug/dwarf.(*Data).readType.func3(0xc01c79a1b0, 0xc05061b800, 0x0)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01834c640, 0x453d363, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:535 +0x518
debug/dwarf.(*Data).readType.func3(0xc01c78de90, 0xc000000049, 0xc05061b560)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01834c5a0, 0x453d3e7, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:644 +0x1f03
debug/dwarf.(*Data).readType.func3(0xc01c78de60, 0xc05061b500, 0x0)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01833d5e0, 0x453e0fb, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:535 +0x518
debug/dwarf.(*Data).readType.func3(0xc01c77aae0, 0xc000000049, 0xc050616b20)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01833d540, 0x453e460, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:644 +0x1f03
debug/dwarf.(*Data).readType.func3(0xc01c77aab0, 0xc050616ac0, 0x0)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc01833d0e0, 0x453e077, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:535 +0x518
debug/dwarf.(*Data).readType.func3(0xc01c77a7e0, 0xc050616880, 0x0)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc0182ede00, 0x453c49e, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:535 +0x518
debug/dwarf.(*Data).readType.func3(0xc01c72b380, 0xc000000049, 0xc04fe39f60)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc0182edd60, 0x453cfb0, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:644 +0x1f03
debug/dwarf.(*Data).readType.func3(0xc01c72b350, 0xc04fe39f00, 0x0)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:379 +0x235
debug/dwarf.(*Data).readType(0xc000092000, 0x55cf54, 0x4, 0x58b000, 0xc0182edc20, 0x453c430, 0xc00007a6c0, 0xc05ae9f770, 0x0, 0x0, ...)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:535 +0x518
debug/dwarf.(*Data).Type(0xc000092000, 0xc00453c430, 0xc02cd88920, 0x18, 0xc0351ee8a0, 0x0)
    /usr/lib/go-1.15/src/debug/dwarf/type.go:292 +0xbe
main.(*vtypeJson).addDwarf.func2(0xc01c72b260, 0x8, 0x0, 0x0)
    /home/shellcode/Tools/dwarf2json/main.go:246 +0x84b
main.(*vtypeJson).addDwarf(0xc00007a900, 0xc000092000, 0x55d42d, 0x6, 0x3, 0xc00000e200, 0x1)
    /home/shellcode/Tools/dwarf2json/main.go:389 +0x2a5
main.generateLinux(0xc00000e200, 0x1, 0x1, 0x0, 0x0, 0x0)
    /home/shellcode/Tools/dwarf2json/main.go:992 +0x8d5
main.main()
    /home/shellcode/Tools/dwarf2json/main.go:703 +0xe11

I'm not familiard with Go, so no idea what's going on, but let me know if you want me to send the vmlinux that causes the crash.

Linux laptop 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux

go version go1.15.15 linux/amd64

EDIT :

Tried in a VM with :

go version go1.19.3 linux/amd64

worked without issue.

Creating Symbol Table for Monterey - Failed validating 'oneOf' in schema

./dwarf2json mac --macho /Library/Developer/KDKs/KDK_12.2.1_21D62.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/DWARF/kernel --macho-symbols /Library/Developer/KDKs/KDK_12.2.1_21D62.kdk/System/Library/Kernels/kernel > monterey_Kernel12.2.1_21D62.json

./dwarf2json mac --macho /Library/Developer/KDKs/KDK_12.2.1_21D62.kdk/System/Library/Kernels/kernel.release.t6000.dSYM/Contents/Resources/DWARF/kernel.release.t6000 --macho-symbols /Library/Developer/KDKs/KDK_12.2.1_21D62.kdk/System/Library/Kernels/kernel.release.t6000 > monterey_Kernel12.2.1_21D62_t6000.json

Compressed json files to the xz format and copied to volatility3/symbols/mac

copied compressed file to volatility3/symbols/mac

Run volatility isfinfo to ensure the new symbol table is recognised

python3 vol.py isfinfo | grep mac
file:///volatility3/framework/symbols/mac/sierra_12.2.1_16G2128.json.xz True (cached) 18 0 43736 166 b'Darwin Kernel Version 16.7.0: Sun Jun 2 20:26:31 PDT 2019; root:xnu-3789.73.501/RELEASE_X86_64\x00'
file:///volatility3/symbols/mac/monterey_Kernel12.2.1_21D62_t6000.json.xz True (cached) 19 0 58919 361 b'Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.2420/RELEASE_ARM64_T6000\x00'
file:///volatility3/symbols/mac/monterey_Kernel12.2.1_21D62.json.xz True (cached) 19 0 62951 369 b'Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64\x00'

Python3 vol.py isfinfo | grep mac

Run the volatility3 banner on memory to ensure the symbol table is a match

python3 vol.py -f OSX_Monterey.lime banners
Volatility 3 Framework 2.4.1
Progress: 100.00 PDB scanning finished
Offset Banner

0x10004aa4bd5 Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.2420/RELEASE_ARM64_T6000
0x10004aa4c3c Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.2420/RELEASE_ARM64_T6000
0x10008516d8c Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.2420/RELEASE_ARM64_T6000
0x1000e9b3100 Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.2420/RELEASE_ARM64_T6000

Finally, run mac.pslist plugin (with -vvvvv).

_DEBUG volatility3.framework.automagic.symbol_cache: Duplicate entry for identifier b'Darwin Kernel Version 16.7.0: Sun Jun 2 20:26:31 PDT 2019; root:xnu-3789.73.501/RELEASE_X86_64\x00': file://volatility3/volatility3/framework/symbols/mac/sierra_12.2.1_16G2127.json.xz and file:///volatility3/volatility3/framework/symbols/mac/sierra_12.2.1_16G2128.json.xz
DEBUG volatility3.framework.automagic.mac: Identified banner: b'Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.2420/RELEASE_ARM64_T6000\x00'
DEBUG volatility3.schemas: Validating JSON against schema...
Level 7 volatility3.framework.automagic.stacker: Exception during stacking: {'count': 33, 'kind': 'array', 'subtype': None} is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['user_types']['additionalProperties']['properties']['fields']['additionalProperties']['properties']['type']:
{'oneOf': [{'$ref': '#/definitions/type_pointer'},
{'$ref': '#/definitions/type_base'},
{'$ref': '#/definitions/type_array'},
{'$ref': '#/definitions/type_struct'},
{'$ref': '#/definitions/type_enum'},
{'$ref': '#/definitions/type_function'},
{'$ref': '#/definitions/type_bitfield'}]}

On instance['user_types']['host']['fields']['special']['type']:
{'count': 33, 'kind': 'array', 'subtype': None}
Level 9 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Lsof.kernel.layer_name
Level 9 volatility3.framework.configuration.requirements: TypeError - Layer is not the required Architecture: LimeLayer
Level 9 volatility3.framework.configuration.requirements: TypeError - Layer is not the required Architecture: FileLayer
DEBUG volatility3.framework.automagic.stacker: Stacked layers: ['LimeLayer', 'FileLayer']
INFO volatility3.framework.automagic: Running automagic: SymbolFinder
INFO volatility3.framework.automagic: Running automagic: MacSymbolFinder
Level 9 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Lsof.kernel.symbol_table_name
INFO volatility3.framework.automagic: Running automagic: KernelModule
Level 9 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Lsof.kernel.layer_name
Level 9 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Lsof.kernel.symbol_table_name
Level 9 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Lsof.kernel.layer_name
Level 9 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Lsof.kernel.layer_name
Level 9 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Lsof.kernel.symbol_table_name_

Any help here would be greatly appreciated

Issues with anonymous types

Dwarf2json is missing information. I noticed it is happening with anonymous types, but it could also happen in other cases.

I used pdwtags from dwarves package to debug this issue.

The following is page struct seen from pdwtags.

Note that I intentionally changed the order of some structs below inside the union so that it is easier to see.

$ pdwtags vmlinux-4.18.0-10-generic  | grep -A80 "struct page {" 
struct page {
	long unsigned int          flags;                /*     0     8 */
	union {           // <-- (*) unnamed_field_8
		struct callback_head callback_head __attribute__((__aligned__(8))); /*     8    16 */
		struct {
			struct list_head lru;            /*     8    16 */
			struct address_space * mapping;  /*    24     8 */
			long unsigned int index;         /*    32     8 */
			long unsigned int private;       /*    40     8 */
		};                                       /*     8    40 */
		struct {
			struct dev_pagemap * pgmap;      /*     8     8 */
			long unsigned int hmm_data;      /*    16     8 */
			long unsigned int _zd_pad_1;     /*    24     8 */
		};     
		struct { ... }; // <-- 4
		struct { ... }; // <-- 5
		struct { ... }; // <-- 6
		struct { ... }; // <-- 7: As per pdwtags there are seven structs in this union.
	} __attribute__((__aligned__(8)));                                               /*     8    40 */
...

Let's see how dwarf2json interprets the same file:

$ ./dwarf2json linux --elf vmlinux-4.18.0-10-generic 
...
    "page": {
      "size": 64,
      "fields": {
        "flags": {
          "type": {
            "kind": "base",
            "name": "long unsigned int"
          },
          "offset": 0
        },
        "unnamed_field_8": {  <-- First union in offset 8
          "type": {
            "kind": "union",
            "name": "unnamed_d47e6a55607cc2da"    <-- (*)
          },
          "offset": 8,
          "anonymous": true
        },
        ...
      },
      "kind": "struct"
    },

Union unnamed_field_8 is defined by unnamed_d47e6a55607cc2da. As per pdwtags, this union contains 7 anonymous structs, however, for some reason dwarf2json only dumps two of them.

    "unnamed_d47e6a55607cc2da": {
      "size": 40,
      "fields": {
        "callback_head": {     <-- (1)
          "type": {
            "kind": "struct",
            "name": "callback_head"
          },
          "offset": 0
        },
        "unnamed_field_0": {     <-- (2)
          "type": {
            "kind": "struct",
            "name": "unnamed_8209487da916be8a"  <-- The one which contains pgmap, hmm_data and _zd_pad_1
          },
          "offset": 0,
          "anonymous": true
        }                                        <-- (???)
      },
      "kind": "union"
    },

As you can see above, there are missing fields.

OS Debug Symbols: Linux / Ubuntu 18.10
Kernel: 4.18.0-10-generic
Kernel debug symbols package: linux-image-unsigned-4.18.0-10-generic-dbgsym

$ sha1sum /usr/lib/debug/boot/vmlinux-4.18.0-10-generic
e6a1f5c2421c7fed9473e0cbe8b4c85d1099e5d5  /usr/lib/debug/boot/vmlinux-4.18.0-10-generic

This is most likely related to issue #11 .

linux-image-4.18.0-10-generic.zip

[Urgent - Please help!] What is the appropriate kernel ELF to use with dwarf2json? [ 5.19.0-42-generic #43~22.04.1-Ubuntu ]

$ uname -a
Linux UbuntuVM 5.19.0-42-generic #43~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Apr 21 16:51:08 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

I need this to create generate a custom symbols table (using dwarf2json), in order to run a memory dump acquired by Ubuntu 22.04, as Ubuntu 22.04 kernel does not work anymore with volatility 2 (Issue here: volatilityfoundation/volatility#828)

If I use the compressed ..ddeb file I get a "bad magic number" error.

sudo ./dwarf2json linux --elf linux-image-5.19.0-41-generic-dbgsym_5.19.0-41.42_amd64.ddeb > kernel.json
[sudo] password for odin: 
Failed linux processing: could not open linux-image-5.19.0-41-generic-dbgsym_5.19.0-41.42_amd64.ddeb: bad magic number '[33 60 97 114]' in record at byte 0x0

The error message I encountered indicates that the dwarf2json tool was unable to open the specified .ddeb file. The reason for this could be that the dwarf2json tool expects an ELF file as input, not a .ddeb file.

The .ddeb files I downloaded are Debian debug symbol packages, and they are not directly compatible with the dwarf2json tool. The tool typically expects uncompressed ELF files.

So, where can I find the uncompressed ELF file for this kernel version? Help me please, it's urgent. Thanks a lot!

Feature request: /proc/kallsyms support

When available, it would be great to take advantage of /proc/kallsyms to replace system.map and get the real addresses. Also, it could be useful for ASLR and to speed up the profile/ISF detection process, get the banner, etc.
Rekall ( see here ) implements that for similar purposes.

Underflow when processing module.ko file generated from an Android kernel (goldfish)

Hey, im having trouble generating the ISF from an Android kernel:

Goldfish 3.18
Arch: x86_64
dwarf2json: linux-module-method branch

I've tried 2 different ways (linux_build_module & vol2/tools/linux folders) to retrieve the module.ko from this kernel. But when trying to generate the ISF using dwarf2json im always getting the following error:

dwarf2json linux --elf linux_build_module/module.ko 
Failed linux processing: error processing DWARF: decoding dwarf section str at offset 0x0: underflow

The Makefile im using to retrieve the module.ko looks as follows:

obj-m += module.o
KDIR := ~/goldfish/
CCPATH := ~/x86_64-linux-android-4.8/bin/

-include version.mk

all: dwarf 

dwarf: module.c
	$(MAKE) ARCH=x86_64 CROSS_COMPILE=$(CCPATH)/x86_64-linux-android- -C $(KDIR) CONFIG_DEBUG_INFO=y M="$(PWD)" modules

Am I doing something wrong here or is this a problem with dwarf2json?
Any help would be greatly appreciated!

Dwarf2json doesn't generate the appropriate banner for mac systems

I dunno if this repo has been kept up-to-date, but it appears not to properly support creating the necessary constant_data field for the version symbol when operating on mac dwarf files, in order to allow volatility to appropriately detect the mac tables. This relates to volatility3 issue #61.

Linux: OOM when a kernel debug symbols is used as input

Dwarf2json out of memory when parsing Linux kernel debug symbols.
The QEMU VM used for this have Ubuntu 18.04.4 amd64 stock kernels 4.18.0-25-generic and I also tried with 5.4.0-42-generic.
The command lines were:
./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-42-generic
or
./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-42-generic --system-map System.map-5.4.0-42-generic

The dmesg error was:

[  148.628383] Out of memory: Killed process 1322 (dwarf2json) total-vm:5022196kB, anon-rss:3736096kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:8012kB oom_score_adj:0

I tried from inside of the virtual machine first with 2GB of RAM which later increased to 4GB and 8GB but it didn't fix the problem.
Finally, I got dwarf2json working in a 16GB RAM host.
I'm aware Dwarf itself requires a lot of memory but I think it would be great to see if this limitation could be addressed in the future.

Dwarf2json can produce bitfield JSON that doesn't adhere to the schema

Hiya,

Having regenerated a bunch of profiles, including some new ones, it appears there are some bad values being produced. With jsonschema installed, trying to use one of these produces:

DEBUG    volatility.framework.automagic.mac: Identified banner: b'Darwin Kernel Version 13.2.0: Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64\x00'
DEBUG    volatility.schemas: Validating JSON against schema...
DEBUG    volatility.schemas: Schema validation error
Traceback (most recent call last):
  File "/home/personal/workspace/volatility3/volatility/schemas/__init__.py", line 68, in valid
    jsonschema.validate(input, schema)
  File "/usr/lib/python3.7/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: None is valid under each of {'$ref': '#/definitions/type_enum'}, {'$ref': '#/definitions/type_base'}

Failed validating 'oneOf' in schema[6]['properties']['type']:
    {'oneOf': [{'$ref': '#/definitions/type_base'},
               {'$ref': '#/definitions/type_enum'}]}

On instance['type']:
    None

which turns out to be because of the followin code generated from dwarf2json:

    "IOExternalAsyncMethod": {
      "size": 48,
      "fields": {
        "count0": {
          "type": {
            "kind": "base",
            "name": "long long unsigned int"
          },
          "offset": 32
        },
        "count1": {
          "type": {
            "kind": "base",
            "name": "long long unsigned int"
          },
          "offset": 40
        },
        "flags": {
          "type": {
            "kind": "base",
            "name": "unsigned int"
          },
          "offset": 24
        },
        "func": {
          "type": {
            "bit_length": 128,
            "bit_position": -128,
            "kind": "bitfield",
            "type": null
          },
          "offset": 8
        },
        "object": {
          "type": {
            "kind": "pointer",
            "subtype": {
              "kind": "class",
              "name": "IOService"
            }
          },
          "offset": 0
        }
      },
      "kind": "struct"
    },

which features a null value (which can be both enum or base). The bit_length and bit_position also look wrong. I've got the original KDK files for this that were used, and I'll attach the generated JSON. Shout if you need any more information... 5:)

Infinite loop?

On Fedora 32, I am trying to construct a json for Volatility 3. I have built dwarf2json version 0.6.0 from https://github.com/volatilityfoundation/dwarf2json under Fedora 32 using golang-1.14.3-1. The build is successful, that is there are no errors. I then installed both kernel-debuginfo-5.6.15-300.fc32.x86_64 and kernel-debuginfo-common-x86_64-5.6.15-300.fc32.x86_64. I ran dwarf2json as follows:

$ dwarf2json-fc32x linux --elf /usr/lib/debug/usr/lib/modules/5.6.15-300.fc32.x86_64/vmlinux --system-map /boot/System.map-5.6.15-300.fc32.x86_64

and I get the errors noted in the attached text file
dwarf2json-error-001.txt

I received a recommendation from the Volatility 3 developer group that felt the issue was with dwarf2json which is why I am posting this note.

Thanks for any help. I have the same problem on Fedora 29 and CentOS 7.8.

volatilityfoundation / dwarf2json Goto Github PK

dwarf2json's People

Contributors

Stargazers

Watchers

Forkers

dwarf2json's Issues

This is most likely related to issue #11 .

Recommend Projects

Recommend Topics

Recommend Org