Hello everyone,
In this post I will be covering how we handled the issues we had after forking to V6, listing down the real issues despite the FUD ones. I wanted to write this post earlier but we were so busy analyzing and discussing how we would handle the issues from last few days that I couldn’t fine a reasonable time window to write a complete post like this one.
Firstly, I would like to list all of our issues after forking to V6 as well as doing a brief commentary of each one.
1) Difficulty was too high considering post-fork hash rate.
Going straight to the point: yes, it was our fault. We didn’t expected such sudden drop in the hash rate after forking (from 1.6GH/s to 2MH/s). To be honest we were predicting a hash rate of ~200MH/s. However, we have acknowledged that our prediction doesn’t make any sense at all! Why would we still have 200MH/s of CPU/GPU miners in such high difficulty set by ASICs? It simply doesn’t make sense and now we understand that.
2) Mempool had some RingSize 5 transactions that were invalidating the block, thus not being able to mine any block at all.
We missed the mempool handling of old transactions, meaning that we should have implemented a code to drop invalid transactions from the mempool according to the current network consensus (transaction max ringsize = 1). This was causing blocks containing these invalid transactions being rejected at node level and the network got stuck, not being able to output a single block.
How we handled these problems in the last few days:
We didn’t wanted to fork again in order to fix these issues. Forking means that all pools, exchanges and miners would need to update their software in a relatively short period of time, and usually an emergency fork is not something good either. Plus, forking would involve few more days of testing, releasing binaries and waiting for everyone to update.
With that in mind, this is how we decided to manage the issues:
Issue #1: we decided to rent some hashing power in NiceHash to speed up the difficulty adjustment process. However, since the network was stuck due to the issue #2, we had to start mining empty blocks because they would not get rejected as they are empty of transactions. Mining empty blocks gave some time to invalid ringsize 5 transactions in the mempool to expire so that we could start mine regular blocks again. We could have switched back to mine regular blocks after 3 days from the fork (transactions have 3 days life time in the mempool), but we didn’t decided to do so. NanoPool was reporting that some blocks were still getting rejected on their end, so we decided to keep mining empty blocks to reduce the difficulty so that, with the block time reduced to a few minutes, we would be able to better investigate the issue reported more efficiently. Plus, going back to mine regular blocks would mean that some blocks could be still rejected (according to NanoPool), thus drastically increasing the time needed to reach block 307575 in which the difficulty level would be adjusted. In addition, if we decided to mine regular blocks at that time and a block gets rejected, we would have spent way more with NiceHash than we currently did in order to reduce the difficulty level, since rejected blocks are not considered by the difficulty adjustment algorithm.
Issue #2: for the invalid transactions in the mempool, we have simply waited for them to expire. However, in parallel, we implemented the proper mempool handling of invalid transactions so it gets flushed automatically. This patch would be released soon and it wont require a fork since it is a minor update. We had the chance to apply this patch to our own daemon when the mempool had invalid transaction in order to fully test this change, it and went fine!
Today we reached block 307575 and the difficulty level have dropped significantly as we predicted. With the difficulty level adjusted within GPU rig ranges, several pools other than Nano (which was mining empty blocks) could successfully mine regular blocks (with transactions included). Meaning that the block rejection issue reported by NanoPool was just “bad luck” from their side. It’s worth mention that we didn’t release or send any additional patches to the pools. They are actually running our ETN V2.0.0 original code and being able to mine regular blocks!
It is worth mentioning that a few blocks are still empty because NanoPool haven’t reverted their code to original ETN V2.0.0. They are running a specific code to mine empty blocks and it should be reverted to normal soon. We may see some more empty blocks until they do this revert. Fairhash also has a specific code regarding mempool treatment in which they are only keeping 5 TXs in it for logging and debugging purposes. They should revert the code to original ETN v2.0.0 and start mining regular blocks soon.
Couple interesting points I noticed today:
One of the blocks that were mined earlier today had a total of 222 transactions in it, meaning that we could support ~2 transactions per second. That is not bad at all compared to other blockchain technologies. Still, we’re looking forward to increase our transaction throughput even more in the future!
Electroneum is currently the most profitable CryptoNight coin to mine!
I would like to thank everyone who contributed supporting us during this tough period.
Some of you even suggested possible fixes on GitHub!
Regards,
Andre Patta
Complete discussion can be found here:
How Electroneum handled issues after forking to V6? from Electroneum