Example usage of WSH under QEMU user-space emulation?

[endrazine]

Example usage of WSH under QEMU user-space emulation?

[endrazine]

Hi Mike,
I'm glad you're looking into using ARM binaries from Intel :)
I have created a wiki page to describe one way to achieve this using a chroot : https://github.com/endrazine/wcc/wiki/wsh-with-ARM

Strictly speaking, you don't need a chroot, but it's convenient to make sure all dependencies are met.

I am not sure what is happening in this case, we would need a bit more output. You could re-run wsh in verbose mode by adding the -v command line option, that would show us how early the segfault happens.

If you have a binutils version of ARM available already, would you mind sharing with me the output of

$ qemu-arm-static -L /usr/arm-linux-gnueabi/ ldd ./wcc/bin/wsh-arm  

where ldd is the ARM binary provided with binutils ? That would help us make sure wsh-arm is linked properly. Hello seems to be linked fine since it ran previously :)

Finally, it might be due to the fact that wsh-arm and hello have overlapping segments. This is why we link wsh using a custom linker script that sets a non standard base address. You can use the readelf command from binutils to verify what is the base address of the .text segment in both binaries:

readelf -l /usr/bin/wsh-arm hello

If they overlap, you may want to recompile wsh-arm with a different base address. Let me know if you need help with this.

[michael-myers]

Adding the -v option to wsh (seems like it fails before reaching any verbose output):

$ qemu-arm-static -L /usr/arm-linux-gnueabi/ ./wcc/bin/wsh-arm -v
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

Checking for overlapping segments (at a glance, they don't seem to overlap):

$ readelf -l ./wcc/bin/wsh-arm hello

File: ./wcc/bin/wsh-arm

Elf file type is EXEC (Executable file)
Entry point 0x19c1c
There are 10 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  EXIDX          0x0fe320 0x00106320 0x00106320 0x00810 0x00810 R   0x4
  PHDR           0x000034 0x00008034 0x00008034 0x00140 0x00140 R E 0x4
  INTERP         0x000174 0x00008174 0x00008174 0x00029 0x00029 R   0x1
      [Requesting program interpreter: /usr/arm-linux-gnueabi/lib/ld-linux.so.3]
  LOAD           0x000000 0x00008000 0x00008000 0xfeb34 0xfeb34 R E 0x8000
  LOAD           0x0ffa2c 0x0010fa2c 0x0010fa2c 0x02d3c 0x043b8 RW  0x8000
  DYNAMIC        0x100f10 0x00110f10 0x00110f10 0x000f0 0x000f0 RW  0x4
  NOTE           0x0001a0 0x000081a0 0x000081a0 0x00044 0x00044 R   0x4
  TLS            0x0ffa2c 0x0010fa2c 0x0010fa2c 0x00010 0x00028 R   0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x10
  GNU_RELRO      0x0ffa2c 0x0010fa2c 0x0010fa2c 0x015d4 0x015d4 R   0x1

 Section to Segment mapping:
  Segment Sections...
   00     .ARM.exidx 
   01     
   02     .interp 
   03     .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rel.dyn .rel.plt .init .plt .text __libc_freeres_fn __libc_thread_freeres_fn .fini .rodata __libc_subfreeres __libc_atexit __libc_thread_subfreeres .ARM.extab .ARM.exidx .eh_frame 
   04     .tdata .init_array .fini_array .jcr .data.rel.ro .dynamic .got .data .bss __libc_freeres_ptrs 
   05     .dynamic 
   06     .note.ABI-tag .note.gnu.build-id 
   07     .tdata .tbss 
   08     
   09     .tdata .init_array .fini_array .jcr .data.rel.ro .dynamic 

File: hello

Elf file type is EXEC (Executable file)
Entry point 0x1030c
There are 9 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  EXIDX          0x0004e0 0x000104e0 0x000104e0 0x00008 0x00008 R   0x4
  PHDR           0x000034 0x00010034 0x00010034 0x00120 0x00120 R E 0x4
  INTERP         0x000154 0x00010154 0x00010154 0x00013 0x00013 R   0x1
      [Requesting program interpreter: /lib/ld-linux.so.3]
  LOAD           0x000000 0x00010000 0x00010000 0x004ec 0x004ec R E 0x10000
  LOAD           0x000f0c 0x00020f0c 0x00020f0c 0x0011c 0x00120 RW  0x10000
  DYNAMIC        0x000f18 0x00020f18 0x00020f18 0x000e8 0x000e8 RW  0x4
  NOTE           0x000168 0x00010168 0x00010168 0x00044 0x00044 R   0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x10
  GNU_RELRO      0x000f0c 0x00020f0c 0x00020f0c 0x000f4 0x000f4 R   0x1

 Section to Segment mapping:
  Segment Sections...
   00     .ARM.exidx 
   01     
   02     .interp 
   03     .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rel.dyn .rel.plt .init .plt .text .fini .rodata .ARM.exidx .eh_frame 
   04     .init_array .fini_array .jcr .dynamic .got .data .bss 
   05     .dynamic 
   06     .note.ABI-tag .note.gnu.build-id 
   07     
   08     .init_array .fini_array .jcr .dynamic

About whether wsh-arm is linked properly:

$ find / -name ldd 2>/dev/null
/usr/bin/ldd

Hmm I have binutils installed by it doesn't include an ARM build of ldd, does the following help?:

$ arm-linux-gnueabi-readelf -a ./wcc/bin/wsh-arm | grep "Shared library:"
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
$ find / -name libgcc_s.so.1 2>/dev/null
/lib/x86_64-linux-gnu/libgcc_s.so.1
/usr/arm-linux-gnueabi/lib/libgcc_s.so.1
/usr/lib/gcc-cross/arm-linux-gnueabi/5/libgcc_s.so.1

It seems like wsh-arm would be able to find the one shared library it needs?

[endrazine]

Hi Mike,

Let's try to get this command to work first:

$ qemu-arm-static -L /usr/arm-linux-gnueabi/ ./wcc/bin/wsh-arm -h

This doesn't do much more than display a help message via printf(), so it's very close to the helloworld.c test case. To work, it will however need the wsh-arm dependencies to be correct. In addition to the C library, wsh also links with libdl. You seem to also have a dependency on libgcc_s.so.1 which is the runtime of the gcc compiler.

The expected output is this:

jonathan@blackbox:~$ qemu-arm-static -L /usr/arm-linux-gnueabi/ /usr/bin/wsh-arm -h
Usage: /usr/bin/wsh-arm [script] [options] [binary1] [binary2] ... [-x] [script_arg1] [script_arg2] ...

Options:

    -x, --args                Optional script argument separator.
    -v, --verbose
    -V, --version

Script:

    If the first argument is an existing file which is not a known binary file format,
    it is assumed to be a lua script and gets executed.

Binaries:

    Any binary file name before the -x tag gets loaded before running the script.
    The last binary loaded is the main binary analyzed.

jonathan@blackbox:~$

To print the immediate shared libraries dependencies (not recursively), you can use objdump, such as in the following command:

jonathan@blackbox:~$ arm-linux-gnueabi-objdump /usr/bin/wsh-arm -x|grep NEEDED
  NEEDED               libdl.so.2
  NEEDED               libc.so.6
  NEEDED               ld-linux-armhf.so.3
jonathan@blackbox:~$ 

If your version of objdump is built with multiarch support (there is a binutils-multiarch package in ubuntu), you can simply use:

jonathan@blackbox:~$ objdump /usr/bin/wsh-arm -x|grep NEEDED
  NEEDED               libdl.so.2
  NEEDED               libc.so.6
  NEEDED               ld-linux-armhf.so.3
jonathan@blackbox:~$

Looking at the outputs of readelf -l for the two ARM executables, it seems to me that wsh-arm might have the wrong dynamic loader path hardcoded in its .interp interpreter segment.

You seem to have the following interpreters hardcoded in the binaries :

wsh-arm --> /usr/arm-linux-gnueabi/lib/ld-linux.so.3
hello --> /lib/ld-linux.so.3

That's not an uncommon problem when working with cross compilation toolchains instead of full chroots. Let's see if you manage to adjust your toolchain to fix this wsh-arm compilation issue.

You can display the content of the .interp section using readelf -l as you did previously, or using the -x flag to display the content in hexadecimal of a given section:

jonathan@blackbox:~$ readelf /usr/bin/wsh-arm -x .interp

Hex dump of section '.interp':
  0x00008154 2f6c6962 2f6c642d 6c696e75 782d6172 /lib/ld-linux-ar
  0x00008164 6d68662e 736f2e33 00                mhf.so.3.

jonathan@blackbox:~$ 

I hope it helps :)

Good luck,
Best regards,
j-

[michael-myers]

Thanks for your suggestions. The launch of wsh-arm with only the -h command line argument experiences the same segfault problem, so I will use it as the test case. You are correct that wsh-arm is building with different linkages than my hello world binary. To begin addressing this I did a make clean and another make all: afterward, I was surprised to find that the ./bin directory has no wsh-arm in it.

I looked back over my bash history at this point. I had never explicitly called the ARM makefile but this time I tried it:

$ make -f Makefile-arm
arm-linux-gnueabi-gcc-4.7  -rdynamic -W -Wall -Wextra -O0 -ggdb -g3 -Wno-unused-but-set-variable -Wno-unused-parameter -I./include/sflib/ -I./include -I../../include/ -rdynamic   wsh.c -o wsh.o -c
make: arm-linux-gnueabi-gcc-4.7: Command not found
Makefile-arm:15: recipe for target 'all' failed
make: *** [all] Error 127

Huh, that's weird – was the wsh-arm binary in an earlier version of the Git repo?

$ cd wcc; git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)

    modified:   bin/wcc
    deleted:    bin/wcc32
    modified:   bin/wld
    deleted:    bin/wld32
    deleted:    bin/wlld
    modified:   bin/wsh
    deleted:    bin/wsh-arm
    deleted:    bin/wsh-i386
    deleted:    bin/wsh32
    modified:   src/wcc/wcc.c
    modified:   src/wsh/lua (untracked content)

Indeed, I am thinking I never actually built wsh-arm. It was a bin that you built, in an earlier version of the repo. So I will start over with trying to troubleshoot why wcc/src/wsh/Makefile-arm is or is not working for my setup.

[michael-myers]

Correct me if I am wrong but:

• could Makefile-arm's reference to arm-linux-gnueabi-gcc-4.7 be replaced with arm-linux-gnueabi-gcc?
• it also makes a reference to /home/jonathan as a runtime path, this seems like a mistake
• I have googled this but still don't understand if I ought to install the arm-linux-gnueabihf package or if I should change these dependency references to point to what I have: arm-linux-gnueabi
• linking with /tmp/libgcc_s.so.1, this also seems like maybe a mistake

[endrazine]

Hi Mike,

was the wsh-arm binary in an earlier version of the Git repo?

You're right, the very first commit included binaries. It was a mistake and I have git rmed them.

Indeed, I am thinking I never actually built wsh-arm.

I would recommend building within a chroot ; while it is technically possible to do without, fixing the requirements manually is going to take way more time (you would need to cross compile all of binutils-dev_2.24-5ubuntu3_armhf.deb libc6-dev-armel-cross_2.15-0ubuntu9cross1.82_all.deb lua5.2_5.2.3-1.1_armhf.deb libiberty-dev_20160215-1_armhf.deb ... that's a lot of work). Once you have all the requirements within your chroot, just use the normal Makefile by typing "make" :)

If you really only want to play with the wsh-arm version you got from the first commit, patching the location of the dynamic linker directly in the binary could work. We'd like to change it from /usr/arm-linux-gnueabi/lib/ld-linux.so.3 to simply /lib/ld-linux.so.3. We can do this with sed :

cat wsh-arm | sed s#"/usr/arm-linux-gnueabi/lib/ld-linux.so.3"#"/lib/ld-linux.so.3\x00usr/arm-linux-gnueabi"#gi >wsh-arm-patched

This way, the dynamic linker will really be /lib/ld-linux.so.3 (asciiz string), the rest of the string is just padding to make sure we do not alter the .interp section size.

I have updated the installation manual with the link to the exact chroot I have been using.
I have also removed the Makefile-arm and Makefile-linaro makefiles so we do not run into this problem again :)

Let me know how this works for you !
Best regards,
j-

[rofl0r]

there should only be one Makefile (not one per compilation flavour), and there should be no hardcoded CC.
it's up to the user to pass CC to the make invocation, e.g.

make CC=armv7hf-linux-musleabi-gcc

likewise, /usr shouldn't be hardcoded. for example on sabotage linux, there is no /usr.
my recommendation would be the following setup:

#set up all variables in this area
#CC doesn't need to be set, it defaults to cc
#CC=gcc
prefix=/usr/local
bindir=$(prefix)/bin

# give the user the opportunity to override these vars by putting them into config.mak
-include config.mak

....

install:
       install -Dm 755 bin/wsh $(DESTDIR)$(bindir)/wsh

this allows the user to override prefix by putting it into config.mak, or to pass it on the command line

make prefix=/usr DESTDIR=foo all install

also: don't use := assignments because this causes immediate expansion. it should only be used for things that eval output of a shell command or similar.
see https://github.com/sabotage-linux/netbsd-curses/blob/master/GNUmakefile as an example (usage is thoroughly described in that repo's README).

[endrazine]

Hi rofl0r, thanks for the suggestion. I'm not sure commenting on thisunrelated bug is the best place though. Would you be kind enough yo submit a patch implementing those enhancements ? I'd be happy to merge it :) Thanks mate !