Update for ongoing bidding bugs

Hey all, this is an update for ongoing bidding bugs due to the reorg on block #58924.

There was a bug with the core Handshake wallet (HSD, not Namebase) that coincidentally

pinheadmz/
had pre-emptively caught and patched quite literally just a week ago. This bug only occurs when you submit a TRANSFER transaction during a blockchain reorg, which happened for the first time ever yesterday on block #58924.

We knew we needed to merge this "transfer reorg" bugfix eventually and were in the middle of testing the updates and changes, which we always do for non-urgent updates, when the "transfer reorg" bug occured. Many of you had experienced this as auction pages were effectively frozen on block #58924. To fix this urgent bug, we skipped some of our QA steps and updated our wallets to the latest HSD version. This fixed the initial problem at hand, but it introduced another unrelated issue for how Handshake addresses are tracked, which we would have caught if we didn't rush through QA before deploying the latest HSD. This new unrelated issue is now causing bids to be stuck in a "SCHEDULED" state, but we know how to fix it and have been on it all of today.

Funds and names are very safe but we cannot send bid transactions while this issue persists, which we estimate should be fixed within the next 4 hours or so.

Super appreciate everyone's patience with this, and once we patch this bug it shouldn't happen again. 🙏

Comments
4
dees/
8 points
•

Thanks for the transparency, this really helps restore trust.

erebus/
4 points
•

Definitely appreciate the transparency here as

dees/
mentioned. Also alongside that bit thank you for working tirelessly to get this resolved. It is appreciated and shows your ongoing commitment to Handshake.

anthony/
4 points
•

We've been successfully sending transactions for the last 7 blocks (2.5 hrs)!

~~ Technical details since I hear from Johnny that some of you are into that ~~

Still investigating the root cause, but current signs suggest the issue occurred as a result of applying the bug fix for the "transfer reorg" bug (

https://github.com/handshake-org/hsd/pull/564/
). Many thanks to
pinheadmz/
for working on the fix for that issue.

Applying this fix involves wiping out the wallet (don't worry, we're very safe and everything was backed up!) and recomputing the wallet state based on the blockchain history (a "wallet rescan"). This is a rather rare need, and only needs to be done in extreme circumstances when there is a serious issue with how the Handshake wallet tracks funds and names. We've done this before, but in a subtly different way that was changed in the latest HSD release.

My current hypothesis (still checking some things) is that minute differences in the deep wallet rescan process caused our wallet to save some addresses incorrectly. A small number (~2 dozen) addresses failed to be saved, which prevented bid transactions from being generated when the resulting addresses were not recognized by the wallet. Note: there was never any risk of fund loss, since even if we sent funds or bids to those addresses (we didn't), they are all deterministically computable via hierarchical derivation (
https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki/
 ).

This issue was compounded by some really nitty gritty complexities with the wallet's lookahead parameter. Handshake has long had a bug (just recently fixed) with how it stores lookaheads (some discussion here
https://github.com/handshake-org/hsd/pull/569/
 ). We also had some custom lookahead stuff, etc etc, gonna skip over the details but basically these two problems compounded and resulted in the bid downtime today. Saving the ~2 dozen mistracked addresses + some lookahead magic seems to have resolved the issue.

dees/
2 points
•

thanks for clarifying, always appreciated! i already have some context on the rescan and lookahead bugs since i uh... helped discover them, let's say. nice work getting all that sorted out rapidly in production.

PS I am very excited to see those blue hyperlinks! however they're all busted!