Problem

[ombre-projects]

Problem

[Elektryon-Dev]

Try this. I think the last commit broke something.
sudo git clone https://github.com/ombre-projects/ombre.git
cd ombre
sudo git checkout e6fcbeb

[Elektryon-Dev]

No.... That doesn't work... Bad idea.

[Elektryon-Dev]

Or maybe? You can try it.

[Bathmat]

Same issue.
Error attempting to retrieve a hard fork version at height 0 from the db: MDB_NOTFOUND: No matching key/data pair found

[mrheinen]

Can you run the daemon with --log-level 4 and give me the result ?

[Bathmat]

This is with checkout e6fcbeb I'll rebuild and get the current version error.

Creating the logger system
info   {1} {p1} 2018-03-05 16:11:30.209197 [main.cpp+288 ::main] Moving from main() into the daemonize now.

2018-Mar-05 16:11:30.209709 Initializing cryptonote protocol...
2018-Mar-05 16:11:30.209913 Cryptonote protocol initialized OK
2018-Mar-05 16:11:30.210155 Blockchain::Blockchain
info   {1} {p1} 2018-03-05 16:11:30.210465 [connection_basic.cpp+166 ::connection_basic] Spawned connection p2p#0 to 0.0.0.0 currently we have sockets count:1
info   {1} {p1} 2018-03-05 16:11:30.210576 [abstract_tcp_server2.inl+85 ::connection] test, connection constructor set m_connection_type=2

2018-Mar-05 16:11:30.210837 Initializing p2p server...
2018-Mar-05 16:11:30.211596 dns_threads[0] created for: seed1.ombre.io
2018-Mar-05 16:11:30.215864 dns_threads created, now waiting for completion or timeout of 20000ms
2018-Mar-05 16:11:30.215883 dns_threads[2] created for: seed3.ombre.io
2018-Mar-05 16:11:30.214833 dns_threads[1] created for: seed2.ombre.io
2018-Mar-05 16:11:30.286512 dns_threads[2] DNS resolve done
2018-Mar-05 16:11:30.287053 dns_threads[2] addr_str: seed3.ombre.io  number of results: 1
2018-Mar-05 16:11:30.286854 dns_threads[1] DNS resolve done
2018-Mar-05 16:11:30.287763 dns_threads[1] addr_str: seed2.ombre.io  number of results: 1
2018-Mar-05 16:11:30.345592 dns_threads[0] DNS resolve done
2018-Mar-05 16:11:30.345813 dns_threads[0] addr_str: seed1.ombre.io  number of results: 1
2018-Mar-05 16:11:30.346168 DNS lookup for seed1.ombre.io: 1 results
2018-Mar-05 16:11:30.346255 DNS lookup for seed2.ombre.io: 1 results
2018-Mar-05 16:11:30.346297 DNS lookup for seed3.ombre.io: 1 results
2018-Mar-05 16:11:30.346341 Seed node: 104.155.77.250:19743
2018-Mar-05 16:11:30.346536 Added seed node: 104.155.77.250:19743
2018-Mar-05 16:11:30.346624 Seed node: 185.101.98.125:19743
2018-Mar-05 16:11:30.346860 Added seed node: 185.101.98.125:19743
2018-Mar-05 16:11:30.347027 Seed node: 185.47.62.242:19743
2018-Mar-05 16:11:30.347169 Added seed node: 185.47.62.242:19743
2018-Mar-05 16:11:30.347300 Number of seed nodes: 3
info   {1} {p1} 2018-03-05 16:11:30.347537 [network_throttle-detail.cpp+166 ::set_target_speed] Setting LIMIT: 4.40402e+06 kbps

2018-Mar-05 16:11:30.348013 Set limit-up to 2048 kB/s
info   {1} {p1} 2018-03-05 16:11:30.348271 [network_throttle-detail.cpp+166 ::set_target_speed] Setting LIMIT: 8.38861e+06 kbps

info   {1} {p1} 2018-03-05 16:11:30.348641 [network_throttle-detail.cpp+166 ::set_target_speed] Setting LIMIT: 8.38861e+06 kbps

2018-Mar-05 16:11:30.348963 Set limit-down to 8192 kB/s
info   {1} {p1} 2018-03-05 16:11:30.349142 [network_throttle-detail.cpp+166 ::set_target_speed] Setting LIMIT: 4.40402e+06 kbps

2018-Mar-05 16:11:30.349324 Set limit-up to 2048 kB/s
info   {1} {p1} 2018-03-05 16:11:30.349484 [network_throttle-detail.cpp+166 ::set_target_speed] Setting LIMIT: 8.38861e+06 kbps

info   {1} {p1} 2018-03-05 16:11:30.349649 [network_throttle-detail.cpp+166 ::set_target_speed] Setting LIMIT: 8.38861e+06 kbps

2018-Mar-05 16:11:30.349787 Set limit-down to 8192 kB/s
info   {1} {p1} 2018-03-05 16:11:30.350086 [abstract_tcp_server2.inl+796 ::set_threads_prefix] Set server type to: 2 from name: P2P

info   {1} {p1} 2018-03-05 16:11:30.350386 [abstract_tcp_server2.inl+797 ::set_threads_prefix] prefix_name = P2P

2018-Mar-05 16:11:30.350569 Binding on 0.0.0.0:19743
FACT    {1} {p1} 2018-03-05 16:11:30.350809 [abstract_tcp_server2.inl+736 ::init_server] start accept

2018-Mar-05 16:11:30.351017 Net service bound to 0.0.0.0:19743
2018-Mar-05 16:11:30.351141 Attempting to add IGD port mapping.
2018-Mar-05 16:11:33.365698 UPnP device was found but not recognized as IGD.
2018-Mar-05 16:11:33.365922 P2p server initialized OK
info   {1} {p1} 2018-03-05 16:11:33.366224 [connection_basic.cpp+166 ::connection_basic] Spawned connection p2p#0 to 0.0.0.0 currently we have sockets count:1
info   {1} {p1} 2018-03-05 16:11:33.366288 [abstract_tcp_server2.inl+85 ::connection] test, connection constructor set m_connection_type=1

2018-Mar-05 16:11:33.366362 Initializing core rpc server...
info   {1} {p1} 2018-03-05 16:11:33.366501 [abstract_tcp_server2.inl+796 ::set_threads_prefix] Set server type to: 1 from name: RPC

info   {1} {p1} 2018-03-05 16:11:33.366586 [abstract_tcp_server2.inl+797 ::set_threads_prefix] prefix_name = RPC

2018-Mar-05 16:11:33.366632 Binding on 127.0.0.1:19744
FACT    {1} {p1} 2018-03-05 16:11:33.366836 [abstract_tcp_server2.inl+736 ::init_server] start accept

2018-Mar-05 16:11:33.366906 Core rpc server initialized OK on port: 19744
2018-Mar-05 16:11:33.366976 Initializing core...
2018-Mar-05 16:11:33.367243 Locking /home/bathmat/.ombre/.daemon_lock
2018-Mar-05 16:11:33.367328 BlockchainLMDB::BlockchainLMDB
2018-Mar-05 16:11:33.367380 BlockchainLMDB::get_db_name
2018-Mar-05 16:11:33.367468 Loading blockchain from folder /home/bathmat/.ombre/lmdb ...
2018-Mar-05 16:11:33.367549 option: fast
2018-Mar-05 16:11:33.367595 option: async
2018-Mar-05 16:11:33.367635 option: 1000
2018-Mar-05 16:11:33.367727 BlockchainLMDB::open
2018-Mar-05 16:11:33.368284 BlockchainLMDB::need_resize
2018-Mar-05 16:11:33.368353 DB map size:     1073741824
2018-Mar-05 16:11:33.368387 Space used:      36864
2018-Mar-05 16:11:33.368417 Space remaining: 1073704960
2018-Mar-05 16:11:33.368462 Size threshold:  0
2018-Mar-05 16:11:33.368638 Percent used: 0.0000  Percent threshold: 0.8000
2018-Mar-05 16:11:33.369520 Setting m_height to: 0
2018-Mar-05 16:11:33.369700 mdb_txn_safe: destructor
2018-Mar-05 16:11:33.369768 Blockchain::init
2018-Mar-05 16:11:33.369850 BlockchainLMDB::height
2018-Mar-05 16:11:33.369898 BlockchainLMDB::check_open
2018-Mar-05 16:11:33.369938 BlockchainLMDB::get_hard_fork_version
2018-Mar-05 16:11:33.370559 BlockchainLMDB::check_open
2018-Mar-05 16:11:33.370874 BlockchainLMDB::block_rtxn_start
2018-Mar-05 16:11:33.371123 Error attempting to retrieve a hard fork version at height 0 from the db: MDB_NOTFOUND: No matching key/data pair found
2018-Mar-05 16:11:33.381564 mdb_txn_safe: destructor
Aborted (core dumped)

[mrheinen]

Thanks! Does this also happen when you remove the .ombre dir? Can you also try with --data-dir /tmp/ ?

[Bathmat]

@SadBatman yep same issues with removing db or using tmp

This was a fresh build today, btw

[mrheinen]

This is my startup log:

2018-Mar-05 17:51:56.462360 Core rpc server initialized OK on port: 19744
2018-Mar-05 17:51:56.462390 Initializing core...
2018-Mar-05 17:51:56.462503 Loading blockchain from folder /home/foo/.ombre/lmdb ...
2018-Mar-05 17:51:56.462519 option: fast
2018-Mar-05 17:51:56.462548 option: async
2018-Mar-05 17:51:56.462558 option: 1000
2018-Mar-05 17:51:56.462756 Error attempting to retrieve a hard fork version at height 0 from the db: MDB_NOTFOUND: No matching key/data pair found
2018-Mar-05 17:51:56.462823 The DB has no hard fork info, reparsing from start
2018-Mar-05 17:51:56.462848 Blockchain not loaded, generating genesis block.
2018-Mar-05 17:51:56.514045 Blockchain initialized. last block: 0, d2070.h11.m51.s56 time ago, current difficulty: 1
2018-Mar-05 17:51:58.040544 Core initialized OK
2018-Mar-05 17:51:58.040581 Starting core rpc server...
2018-Mar-05 17:51:58.040592 Run net_service loop( 2 threads)...
2018-Mar-05 17:51:58.040686 [SRV_MAIN]Core rpc server started ok
2018-Mar-05 17:51:58.040801 [SRV_MAIN]Starting p2p net loop...
2018-Mar-05 17:51:58.040837 [SRV_MAIN]Run net_service loop( 10 threads)...
2018-Mar-05 17:51:59.041060 [P2P1]

It's the same except that the process doesn't get killed. How much memory do you have ? Could you run the process with:

gdb ./ombred
r

And share the output?

[Bathmat]

This machine has 2GB, but it has been running sumokoind fine. (It wasn't running during this, btw)

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./ombred...done.
(gdb) r
Starting program: /usr/local/src/ombre/bin/ombred
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Creating the logger system
2018-Mar-05 17:02:31.136301 Initializing cryptonote protocol...
2018-Mar-05 17:02:31.136572 Cryptonote protocol initialized OK
2018-Mar-05 17:02:31.137067 Initializing p2p server...
[New Thread 0x7ffff26c1700 (LWP 2835)]
[New Thread 0x7ffff1ec0700 (LWP 2836)]
[New Thread 0x7ffff16bf700 (LWP 2837)]
[Thread 0x7ffff16bf700 (LWP 2837) exited]
[Thread 0x7ffff26c1700 (LWP 2835) exited]
2018-Mar-05 17:02:31.330599 Set limit-up to 2048 kB/s
2018-Mar-05 17:02:31.330916 Set limit-down to 8192 kB/s
2018-Mar-05 17:02:31.331180 Set limit-up to 2048 kB/s
2018-Mar-05 17:02:31.331402 Set limit-down to 8192 kB/s
2018-Mar-05 17:02:31.331727 Binding on 0.0.0.0:19743
2018-Mar-05 17:02:31.332733 Net service bound to 0.0.0.0:19743
2018-Mar-05 17:02:31.332957 Attempting to add IGD port mapping.
[Thread 0x7ffff1ec0700 (LWP 2836) exited]
2018-Mar-05 17:02:34.347059 UPnP device was found but not recognized as IGD.
2018-Mar-05 17:02:34.347237 P2p server initialized OK
2018-Mar-05 17:02:34.347400 Initializing core rpc server...
2018-Mar-05 17:02:34.347475 Binding on 127.0.0.1:19744
2018-Mar-05 17:02:34.347635 Core rpc server initialized OK on port: 19744
2018-Mar-05 17:02:34.347689 Initializing core...
2018-Mar-05 17:02:34.347964 Loading blockchain from folder /home/bathmat/.ombre/lmdb ...
2018-Mar-05 17:02:34.348051 option: fast
2018-Mar-05 17:02:34.348090 option: async
2018-Mar-05 17:02:34.348115 option: 1000
2018-Mar-05 17:02:34.348753 Error attempting to retrieve a hard fork version at height 0 from the db: MDB_NOTFOUND: No matching key/data pair found

Thread 1 "ombred" received signal SIGABRT, Aborted.
0x00007ffff50e5428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

[mrheinen]

Could you check the logs under /var/log ? (grep for ombred) and see if there is anything weird ?
Also you have enough space on the drive ?

(I'm trying to reproduce this but am unable to so any information you have additionally helps).

What would be a huge help is if you could:

1. recompile the binaries with -g
2. run the process again with gdb
3. type "bt"

and share the output

[mrheinen]

Btw I'm building debug binaries so you can use these if you wait 10 mins

[mrheinen]

Please try with:
https://github.com/ombre-projects/ombre/releases/download/1.1/ombre-linux-x64-dbg.tar-bz2

Run in gdb and type "bt". Hopefully this gives us more information. Still also check /var/log for entries that indicate the system kills the process.

Cheers!

[Bathmat]

So bt gave:

(gdb) bt
No stack.

However, just running the 1.1 version seems to work fine, but only showing 1 day of blocks?

Sync data returned a new top block candidate: 1 -> 1836 [Your node is 1835 blocks (1 days) behind]

EDIT: Just realized that this ^ is accurate and it is actually the first day.

Also, now that the 1.1 version worked, the previous version seems to be working. Until I delete the db folder, then it's back to the same error.

/var/log shows several of this:
apport.log:ERROR: apport (pid 5990) Mon Mar 5 16:41:50 2018: executable: /usr/local/src/ombre/bin/ombred (command line "./ombred")

And lastly, it didn't have a -g flag for make

[mrheinen]

Argh I hate these kind of issues haha. I mean issues where it's hard to reproduce and suddenly it goes away. Let's keep this open ?

[Bathmat]

Sorry, I made a bunch of edits to my previous post as I was getting information/thinking aloud.

But, the problem is still there if I use the current version and build from source.

[TOGM]

https://pastebin.com/XT0wq1X8

latest commit/master

[Bathmat]

I created a new user to have the daemon run as a service and had the issue show up again. The only way to get it to work was to run the "dbg" version to at least get the first block, then the latest commit/master works. Even with that, it seems to crash frequently.

EDIT: Also I'm seeing this while syncing:

Sync data returned a new top block candidate: 2129 -> 0 [Your node is 2129 blocks (1 days) ahead]

EDIT: Also wallet-rpc seems to be having a core dump using latest commit. Works okay with dbg version.

[mrheinen]

Everyone here is using Ubuntu 16.04 ? I'll setup a VM with that today and will try to reproduce it.

[Bathmat]

Yep. 16.04 server.

[TOGM]

16.04

[mrheinen]

I created an Ubuntu 16.04 virtual machine in 3 configurations:

• 1 CPU and 3.5 GB ram
• 1 CPU and 2 GB ram
• 2 CPU and 2 GB ram

In all cases ombred worked fine (the latest release and non-dbg one) which really is a bummer. I did find a monero pull request that seems related and I'll look into integrating it.

[TOGM]

issue is running:

sudo git clone https://github.com/ombre-projects/ombre.git
cd ombre
sudo git checkout
sudo cmake . 
sudo make -j2

No errors in the build process but will get Error attempting to retrieve a hard fork version at height 0 from the db: MDB_NOTFOUND: No matching key/data pair found

if you run:

sudo git clone https://github.com/ombre-projects/ombre.git
cd ombre
sudo git checkout
sudo make release-static

builds successfully and sync starts happening.

[Bathmat]

I wonder if it's an issue with having Sumokoin on the same system? I know that my system has Sumokoin installed.

[TOGM]

I did mine from a clean install, I don't think that's the issue

[mrheinen]

Could someone give a GDB backtrace of the crash ? Also if anyone could give me access to a syste, where this occurs than that would be great (ping me on telegram)

[Bathmat]

I did a fresh git clone, deleted the DB, make release-static and it started syncing without issues.

[mrheinen]

https://pics.me.me/its-magic-memegen-com-19255903.png ;-)

[Bathmat]

Seeing a new error after fresh installs with make release-static and syncing the daemon in a new DB folder:

2018-Mar-08 18:12:59.812584 [P2P0]ERROR /usr/local/src/ombre/src/cryptonote_core/checkpoints.cpp:107 CHECKPOINT FAILED FOR HEIGHT 5000. EXPECTED HASH: <e838c077bc66356d9bb321d4eb60f0851ef766f0619ddc4c6568a0f149aacea0>, FETCHED HASH: <8ab89dd795c4bd6985902cb9caa6a4289eea9bb63b6463dd2a628a020285439b>
2018-Mar-08 18:12:59.812852 [P2P0]ERROR /usr/local/src/ombre/src/cryptonote_core/blockchain.cpp:3378 WARNING: local blockchain failed to pass a SumoPulse checkpoint, and you could be on a fork. You should either sync up from scratch, OR download a fresh blockchain bootstrap, OR enable checkpoint enforcing with the --enforce-dns-checkpointing command-line option

Trying --enforce-dns-checkpointing flag now.

[Elektryon-Dev]

Fixed in latest commit.

[Bathmat]

Did this also fix the cmake ., make issue?

EDIT: Just tried it and so far no.