running gramine-direct /geth fails, geth runs without gramine

[flashbots]

running gramine-direct /geth fails, geth runs without gramine

[pbeza]

Nope, we can close it. Thanks for asking.

[pbeza]

I can confirm the problem:

ubuntu@ubuntu:~/geth-sgx-gramine$ gramine-direct ./geth
[GETH_INIT] User requested RA-TLS attestation but cannot read SGX-specific file /dev/attestation/attestation_type
[GETH_INIT] creating RA-TLS attestation certificate failed. Aborting...

Perhaps the patched version of flashbots' geth assumes that it will always be run with gramine-sgx (instead of gramine-direct)? If so, there is yet another problem:

ubuntu@ubuntu:~/geth-sgx-gramine$ gramine-sgx ./geth
Gramine is starting. Parsing TOML manifest file, this may take some time...
Killed

Probably because of high memory usage... /var/log/syslog:

localhost systemd[1]: cron.service: A process of this unit has been killed by the OOM killer.

Update

I've just realized that I compiled the project with:

make SGX=1 TLS=1 ENCLAVE_SIZE=64G SEPOLIA=1

as suggested in README.md. I just needed to replace ENCLAVE_SIZE=64G with ENCLAVE_SIZE=16G (I have 32 GB of RAM) to workaround that issue and hit yet another one:

ubuntu@ubuntu:~/geth-sgx-gramine$ gramine-sgx ./geth                                                                                                                                                                                    │Aug 24 19:10:58 localhost node[1009]: 2023-08-24 19:10:58.252 [info]: Request-ID is : undefined
Gramine is starting. Parsing TOML manifest file, this may take some time...                                                                                                                                                                    │Aug 24 19:10:58 localhost node[1009]: 2023-08-24 19:10:58.253 [error]: Intel PCS server returns error(401).{ "statusCode": 401, "message": "Access denied due to invalid subscription key. Make sure you use valid one or no subscription key 
error: AESM service returned error 44; this may indicate that infrastructure for the DCAP attestation requested by Gramine is missing on this machine                                                                                          │at all." }
error: load_enclave() failed with error: Operation not permitted (EPERM)

More details from /var/log/syslog:

node[1009]: 2023-08-24 19:10:57.072 [info]: Client Request-ID : 3f5<TRUNCATED>b43
node[1009]: 2023-08-24 19:10:58.252 [info]: Request-ID is : undefined
node[1009]: 2023-08-24 19:10:58.253 [error]: Intel PCS server returns error(401).{ "statusCode": 401, "message": "Access denied due to invalid subscription key. Make sure you use valid one or no subscription key at all." }
node[1009]: 2023-08-24 19:10:58.253 [error]: Intel PCS server returns error. Error code : 401
node[1009]: 2023-08-24 19:10:58.253 [error]: Error: No cache data for this platform.
node[1009]:     at Module.getPckCertFromPCS (file:///opt/intel/sgx-dcap-pccs/services/logic/commonCacheLogic.js:92:11)
node[1009]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
node[1009]:     at async LazyCachingMode.getPckCertFromPCS (file:///opt/intel/sgx-dcap-pccs/services/caching_modes/cachingMode.js:126:12)
node[1009]:     at async Module.getPckCert (file:///opt/intel/sgx-dcap-pccs/services/pckcertService.js:115:16)
node[1009]:     at async getPckCert (file:///opt/intel/sgx-dcap-pccs/controllers/pckcertController.js:77:25)
aesm_service[1281]: [QPL] No certificate data for this platform.
aesm_service[1281]: [get_platform_quote_cert_data ../qe_logic.cpp:388] Error returned from the p_sgx_get_quote_config API. 0xe011
node[1009]: 2023-08-24 19:10:58.257 [info]: 127.0.0.1 - - [24/Aug/2023:11:10:58 +0000] "GET /sgx/certification/v4/pckcert?qeid=12CE<TRUNCATED>FE6&cpusvn=0B0<TRUNCATED>000&pcesvn=0E00&pceid=0000 HTTP/1.1" 404 32 "-" "-"
CRON[4498]: (root) CMD (flock -xn /tmp/stargate.lock -c '/usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &')

Also, I needed to replace "use_secure_cert": true with "use_secure_cert": false in /etc/sgx_default_qcnl.conf as there was yet another attestation-related error.

[pbeza]

I believe the above error may be due to the PCCS misconfiguration as the RemoteAttestation code sample from sgx-sdk is failing too (as opposed to LocalAttestation that runs just fine):

ubuntu@ubuntu:/opt/intel/sgxsdk/SampleCode/RemoteAttestation$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD/sample_libcrypto ./app

First round, we will try ECDSA algorithm.

Call sgx_get_extended_epid_group_id success.
MSG0 body generated -
4 bytes:
{
0x0, 0x0, 0x0, 0x0
}

Sending msg0 to remote attestation service provider.

Sent MSG0 to remote attestation service.

Call sgx_select_att_key_id success.
Call sgx_create_enclave success.
Call enclave_init_ra success.
Error, call sgx_ra_get_msg1_ex fail [main].
Call enclave_ra_close success.
Second round, we will try EPID algorithm.

Call sgx_get_extended_epid_group_id success.
MSG0 body generated -
4 bytes:
{
0x0, 0x0, 0x0, 0x0
}

Sending msg0 to remote attestation service provider.

Sent MSG0 to remote attestation service.

Call sgx_select_att_key_id success.
Call sgx_create_enclave success.
Call enclave_init_ra success.
Error, call sgx_ra_get_msg1_ex fail [main].
Call enclave_ra_close success.
Enter a character before exit ...

ubuntu@ubuntu:/opt/intel/sgxsdk/SampleCode/RemoteAttestation$ systemctl status aesmd.service
● aesmd.service - Intel(R) Architectural Enclave Service Manager
      Loaded: loaded (/lib/systemd/system/aesmd.service; enabled; vendor preset: enabled)
      Active: active (running) since Thu 2023-08-24 18:59:19 CST; 3 days ago
    Main PID: 1281 (aesm_service)
      Tasks: 4 (limit: 36847)
      Memory: 17.4M
        CPU: 110ms
      CGroup: /system.slice/aesmd.service
              └─1281 /opt/intel/sgx-aesm-service/aesm/aesm_service

Aug 24 18:59:19 ubuntu aesm_service[1281]: The server sock is 0x55e7112b4c70
Aug 24 19:10:58 ubuntu aesm_service[1281]: [QPL] No certificate data for this platform.
Aug 24 19:10:58 ubuntu aesm_service[1281]: [get_platform_quote_cert_data ../qe_logic.cpp:388] Error returned f>
Aug 28 16:53:45 ubuntu aesm_service[1281]: [QPL] No certificate data for this platform.
Aug 28 16:53:45 ubuntu aesm_service[1281]: [get_platform_quote_cert_data ../qe_logic.cpp:388] Error returned f>
Aug 28 16:53:48 ubuntu aesm_service[1281]: [ADMIN]EPID Provisioning initiated
Aug 28 16:53:49 ubuntu aesm_service[1281]: The Request ID is d93ec5d292df42ada3a0c30487aaeab5
Aug 28 16:53:50 ubuntu aesm_service[1281]: The Request ID is 51a7cbc0fffa47c0bec4c48ff21e944b
Aug 28 16:53:50 ubuntu aesm_service[1281]: [ADMIN]EPID Provisioning protocol error reported by Backend (6)
Aug 28 16:53:50 ubuntu aesm_service[1281]: [ADMIN]EPID Provisioning failed

(similar issue here?)

Any help would be appreciated!

[pbeza]

I've just followed this Intel's tutorial on how to set up Intel DCAP and RemoteAttestation sample seems to be working now! (at least ECDSA algorithm, as opposed to the EPID algorithm)

ubuntu@ubuntu:/opt/intel/sgxsdk/SampleCode/RemoteAttestation$ ./app

First round, we will try ECDSA algorithm.

Call sgx_get_extended_epid_group_id success.
MSG0 body generated -
4 bytes:
{
0x0, 0x0, 0x0, 0x0
}

Sending msg0 to remote attestation service provider.

Sent MSG0 to remote attestation service.

Call sgx_select_att_key_id success.
Call sgx_create_enclave success.
Call enclave_init_ra success.
Call sgx_ra_get_msg1_ex success.

MSG1 body generated -
68 bytes:
{

<TRUNCATED>

        Attestation Report:
        id: 0x12345678.
        status: 0.
        revocation_reason: 0.
        pse_status: 0.

        Enclave Report:
        Signature Type: 0x2
        Signature Basename: L
jͬB{
        attributes.flags: 0x7
        attributes.xfrm: 0xe7
        mr_enclave: fe403b95c7dad9d2f3cf15b4ffde1e830bc38b6b3c265dd80d8db70e314e29a2
        mr_signer: 488b27e9d7e55b3df6e44506feb96d19d2ee9bd3f461c409cbd70a92875c9a10
        isv_prod_id: 0x0
        isv_svn: 0x0

Sent MSG3 successfully. Received an attestation result message back

<TRUNCATED>

Secret successfully received from server.
Remote attestation success!
Call enclave_ra_close success.
Second round, we will try EPID algorithm.

Call sgx_get_extended_epid_group_id success.
MSG0 body generated -
4 bytes:
{
0x0, 0x0, 0x0, 0x0
}

Sending msg0 to remote attestation service provider.

Sent MSG0 to remote attestation service.

Call sgx_select_att_key_id success.
Call sgx_create_enclave success.
Call enclave_init_ra success.
Error, call sgx_ra_get_msg1_ex fail [main].
Call enclave_ra_close success.
Enter a character before exit ...

On the other hand geth-sgx-gramine is still failing for some (yet another) reason:

ubuntu@ubuntu:~/geth-sgx-gramine$ gramine-sgx ./geth
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)0x12345678.
                                                                                                                       0.
Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

[GETH_INIT]
[GETH_INIT]   . Creating the RA-TLS server cert and key (using "dcap" as attestation type)...
[GETH_INIT] ok
Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API.
INFO [08-28|19:41:23.588] Starting Geth on Sepolia testnet...
INFO [08-28|19:41:23.589] Maximum peer count                       ETH=50 LES=0 total=50                               fe403b95c7dad9d2f3cf15b4ffde1e830bc38b6b3c265dd80d8db70e314e29a2
INFO [08-28|19:41:23.590] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[08-28|19:41:23.594] Failed to start filesystem watcher       err="function not implemented"
Fatal: Invalid algo in --miner.algotype:

Anyway, that sounds like progress!

[fnerdman]

@pbeza Great that you could figure out the attestation part on your own!

Perhaps the patched version of flashbots' geth assumes that it will always be run with gramine-sgx (instead of gramine-direct)?

Yes, the way the make command is configured, it adds attestation by default, which is not part of gramine-direct.

I just needed to replace ENCLAVE_SIZE=64G with ENCLAVE_SIZE=16G (I have 32 GB of RAM) to workaround that issue

For Sepolia you will need 64GB, the whole DB will be kept in Memory, and geth will OOM during sync with only 16GB. You can add a sufficiently large swap file to prevent OOM issues.

Are you running on an SGX1 or SGX2 machine?

Invalid algo in --miner.algotype

Should be fixed in latest commit.

[pbeza]

Are you running on an SGX1 or SGX2 machine?

SGX2 (/proc/cpuinfo prints Intel(R) Xeon(R) Platinum 8374B CPU @ 2.70GHz)

ubuntu@ubuntu:~/geth-sgx-gramine$ cpuid | grep -i sgx2
      SGX2 supported                           = true
      SGX2 supported                           = true
      SGX2 supported                           = true
      SGX2 supported                           = true
      SGX2 supported                           = true
      SGX2 supported                           = true
      SGX2 supported                           = true
      SGX2 supported                           = true

Invalid algo in --miner.algotype

Should be fixed in latest commit.

Great, thank you for fixing that @lead4good!

Now, after recompiling the project and running gramine-sgx ./geth I'm getting this (any idea?):

ubuntu@ubuntu:~/geth-sgx-gramine$ gramine-sgx ./geth
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

[GETH_INIT]
[GETH_INIT]   . Creating the RA-TLS server cert and key (using "dcap" as attestation type)...
[GETH_INIT] ok
Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API.
INFO [08-29|20:32:01.368] Starting Geth on Sepolia testnet...
INFO [08-29|20:32:01.368] Maximum peer count                       ETH=50 LES=0 total=50
INFO [08-29|20:32:01.370] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[08-29|20:32:01.373] Failed to start filesystem watcher       err="function not implemented"
INFO [08-29|20:32:01.376] Set global gas cap                       cap=50,000,000
INFO [08-29|20:32:01.379] Allocated trie memory caches             clean=154.00MiB dirty=256.00MiB
INFO [08-29|20:32:01.380] Using leveldb as the backing database
INFO [08-29|20:32:01.380] Allocated cache and file handles         database=/root/.ethereum/sepolia/geth/chaindata cache=512.00MiB handles=32768
INFO [08-29|20:32:01.687] Using LevelDB as the backing database
INFO [08-29|20:32:01.704] Opened ancient database                  database=/data/ancient/chain                    readonly=false
INFO [08-29|20:32:01.704] Disk storage enabled for ethash caches   dir=/root/.ethereum/sepolia/geth/ethash count=3
INFO [08-29|20:32:01.705] Disk storage enabled for ethash DAGs     dir=/root/.ethash                       count=2
INFO [08-29|20:32:01.705] Initialising Ethereum protocol           network=11,155,111 dbversion=<nil>
INFO [08-29|20:32:01.705] Writing custom genesis block
INFO [08-29|20:32:01.713] Persisted trie from memory database      nodes=19 size=2.93KiB time=6.484ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [08-29|20:32:01.714]
INFO [08-29|20:32:01.714] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-29|20:32:01.715] Chain ID:  11155111 (sepolia)
INFO [08-29|20:32:01.715] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [08-29|20:32:01.715]
INFO [08-29|20:32:01.715] Pre-Merge hard forks (block based):
INFO [08-29|20:32:01.715]  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [08-29|20:32:01.715]  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [08-29|20:32:01.715]  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [08-29|20:32:01.715]  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [08-29|20:32:01.715]  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [08-29|20:32:01.715]  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [08-29|20:32:01.715]  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [08-29|20:32:01.715]  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [08-29|20:32:01.715]  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [08-29|20:32:01.715]  - Berlin:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [08-29|20:32:01.715]  - London:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [08-29|20:32:01.715]
INFO [08-29|20:32:01.715] Merge configured:
INFO [08-29|20:32:01.715]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [08-29|20:32:01.715]  - Network known to be merged: true
INFO [08-29|20:32:01.715]  - Total terminal difficulty:  17000000000000000
INFO [08-29|20:32:01.715]  - Merge netsplit block:       #1735371
INFO [08-29|20:32:01.715]
INFO [08-29|20:32:01.716] Post-Merge hard forks (timestamp based):
INFO [08-29|20:32:01.716]  - Shanghai:                    @1677557088 (https://github.com/ethereum/execution-specs/blob
/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [08-29|20:32:01.716]
INFO [08-29|20:32:01.716] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-29|20:32:01.716]
INFO [08-29|20:32:01.716] Loaded most recent local block           number=0 hash=25a5cc..3e6dd9 td=131,072 age=1y11mo4d
WARN [08-29|20:32:01.716] Failed to load snapshot                  err="missing or corrupted snapshot"
INFO [08-29|20:32:01.716] Rebuilding state snapshot
INFO [08-29|20:32:01.717] Resuming state snapshot generation       root=5eb6e3..a3f494 accounts=0 slots=0 storage=0.00B dangling=0 elapsed="458µs"
INFO [08-29|20:32:01.717] Regenerated local transaction journal    transactions=0 accounts=0
INFO [08-29|20:32:01.717] Chain post-merge, sync via beacon client
ERROR[08-29|20:32:01.718] Error parsing builder signing key from env err="invalid length, need 256 bits"
ERROR[08-29|20:32:01.718] Builder tx signing key is not set
INFO [08-29|20:32:01.718] new worker                               builderCoinbase=0x0000000000000000000000000000000000000000
ERROR[08-29|20:32:01.718] Builder tx signing key is not set
INFO [08-29|20:32:01.718] new worker                               builderCoinbase=0x0000000000000000000000000000000000000000
ERROR[08-29|20:32:01.718] Builder tx signing key is not set
INFO [08-29|20:32:01.718] new worker                               builderCoinbase=0x0000000000000000000000000000000000000000
ERROR[08-29|20:32:01.718] Builder tx signing key is not set
INFO [08-29|20:32:01.718] new worker                               builderCoinbase=0x0000000000000000000000000000000000000000
INFO [08-29|20:32:01.718] creating multi worker                    config.MaxMergedBundles=3 workers=4
INFO [08-29|20:32:01.718] Gasprice oracle is ignoring threshold set threshold=2
INFO [08-29|20:32:01.718] Flashbots bundle                         ethToCoinbase=0 gasUsed=0 bundleScore=<nil> bundleLength=0 numBundles=0 worker=1
WARN [08-29|20:32:01.719] Error reading unclean shutdown markers   error="leveldb: not found"
INFO [08-29|20:32:01.719] Flashbots bundle                         ethToCoinbase=0 gasUsed=0 bundleScore=<nil> bundleLength=0 numBundles=0 worker=2
INFO [08-29|20:32:01.719] Flashbots bundle                         ethToCoinbase=0 gasUsed=0 bundleScore=<nil> bundleLength=0 numBundles=0 worker=3
WARN [08-29|20:32:01.719] Engine API enabled                       protocol=eth
INFO [08-29|20:32:01.719] requesting                               currentSlot=0
INFO [08-29|20:32:01.719] Generated state snapshot                 accounts=15 slots=0 storage=722.00B dangling=0 elapsed=2.856ms
INFO [08-29|20:32:02.402] Updated validators                       count=23 slot=0
INFO [08-29|20:32:02.402] db dsn is not provided, starting nil db svc
INFO [08-29|20:32:02.403] Start bundle fetcher
INFO [08-29|20:32:02.403] Start receiving mev bundles
INFO [08-29|20:32:02.403] Starting peer-to-peer node               instance=Geth/v1.11.5-stable-03ed9315/linux-amd64/go1.21.0
INFO [08-29|20:32:02.409] New local node record                    seq=1,693,312,322,406 id=edc1ba72619f36bb ip=127.0.0.1 udp=30303 tcp=30303
INFO [08-29|20:32:02.409] Started P2P networking                   self=enode://4a14b2d4631030d46efd4063b82130f3abb62f25cd4ff5820933937b6a98ea2add55a74da796a247f7eaa37f65b4c098f5ac6285db4755d1da9bd170adeb1f02@127.0.0.1:30303
INFO [08-29|20:32:02.411] IPC endpoint opened                      url=/root/.ethereum/sepolia/geth.ipc
INFO [08-29|20:32:02.411] IPC endpoint closed                      url=/root/.ethereum/sepolia/geth.ipc
ERROR[08-29|20:32:04.374] Failed to start filesystem watcher       err="function not implemented"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x709efe]

goroutine 150 [running]:
github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).Close(0x3283940e0)
        github.com/syndtr/[email protected]/leveldb/storage/file_storage.go:574 +0x13e
github.com/syndtr/goleveldb/leveldb.(*DB).Close(0x32851f5c0?)
        github.com/syndtr/[email protected]/leveldb/db.go:1195 +0x332
github.com/ethereum/go-ethereum/p2p/enode.(*DB).Close(0x32840a860)
        github.com/ethereum/go-ethereum/p2p/enode/nodedb.go:500 +0x2c
github.com/ethereum/go-ethereum/p2p.(*Server).run(0x328475180)
        github.com/ethereum/go-ethereum/p2p/server.go:813 +0x143f
created by github.com/ethereum/go-ethereum/p2p.(*Server).Start in goroutine 1
        github.com/ethereum/go-ethereum/p2p/server.go:491 +0x4a8

Relevant patch causing this? (just blindly guessing)

[pbeza]

Just for sake of completeness – the above problem affects Docker-based environment too. You can easily reproduce by running:

docker build --build-arg SEPOLIA=1 --build-arg ENCLAVE_SIZE=16G -t geth-sgx .
docker-compose run geth-sgx

(however, I'm not sure if this is the "canonical" way to build and run this project – README doesn't explain this)

Note that I am using 16G RAM (ENCLAVE_SIZE=16G) because my machine is limited to 32G RAM + I have other services running, but I don't think low RAM is the root cause of the problem. Otherwise I would get an OOM error (no?).

[fnerdman]

@pbeza I can't reproduce the issue you are having.

My guess is that this is a file permission error and gramine isn't forwarding the error properly. E.g. look at this:

INFO [08-29|20:32:02.411] IPC endpoint opened                      url=/root/.ethereum/sepolia/geth.ipc
INFO [08-29|20:32:02.411] IPC endpoint closed                      url=/root/.ethereum/sepolia/geth.ipc

Why is the endpoint closed immediately?

So please:

1. See that geth in SGX has appropriate permissions to access data on disk ( you can verify what folders are accessed via the geth.manifest.template )
2. If you can get geth to sync, restart with 64GB and add a sufficient large SWAP space to your system, so that SGX can swap out in excess of your 32GB system memory.

[pbeza]

My guess is that this is a file permission error and gramine isn't forwarding the error properly. E.g. look at this:

INFO [08-29|20:32:02.411] IPC endpoint opened                      url=/root/.ethereum/sepolia/geth.ipc
INFO [08-29|20:32:02.411] IPC endpoint closed                      url=/root/.ethereum/sepolia/geth.ipc

Why is the endpoint closed immediately?

So please:

1. See that geth in SGX has appropriate permissions to access data on disk ( you can verify what folders are accessed via the geth.manifest.template )

If it was a permissions issue, I think running docker-compose with sudo would solve the problem:

sudo docker-compose run geth-sgx

(but it doesn't)

@lead4good I forgot to mention that I had to comment out this line of docker-compose.yml to be able to run it with docker-compose run geth-sgx (as that file is missing in the repo):

geth-sgx-gramine/docker-compose.yml

Lines 11 to 12 in f2c77c0

	env_file:
	- builder.env

Perhaps this is the root cause of the problem? Can you clarify if I need the builder.env file and where it comes from? Do I need to manually create it and define some environment variables in there to be able to run sgx-geth successfully? I see that there are some environment variables listed in the README , but I (wrongly?) assumed that they were assigned with some default values that allow sgx-geth to run.

[fnerdman]

@pbeza do you still need help resolving this issue?