RT @danrobinson: A gripping read, especially for anyone who enjoyed our Dark Forest post.
@samczsun discovered an exploit that left him in the same position @gakonst and I had been in—but with 1000x as much at stake.
A global team worked together to succeed where we'd failed. https://t.co/syfTDSWsrs
RT @PhABCD: "I tested SparkPool’s whitehat endpoint with a meaningless transaction, and it worked as expected: the tx was not seen in the mempool, then suddenly appeared as part of a SparkPool block!
It was like watching water vapor turn directly into ice without that pesky liquid phase." https://t.co/gnIenHM7m1
alex van de sande (4 character handle ? me : bot) - @avsa3 weeks ago
The post reads like a heist movie, including the long “getting the gang together” part, beautifully written from multiple perspectives.
And yes, this is the a message you don’t want to be in the receiving end of.. https://t.co/48En6dKevqhttps://t.co/asp194djMj
So, a team of hackers colluded with a mining pool to steal ten million dollars' worth of mETH from its rightful owners? The rightful owners, of course, being whoever exploited the vulnerable contract first, since code is law.
See, Ethereum has a ~~bug~~ ~~exploit~~ *undocumented feature*. Instead of taking transactions in chronological order, ~~like Bitcoin~~ (edit: Bitcoin does it too), Eth lets you pay miners extra money to run your transaction first. But transactions on Eth become public *before* they're run by miners. This means, if you're trading, eg, Bitcoin for Eth on an exchange, and someone recognizes you're going to make a profit on that trade, they can copy your transaction with a *higher* transaction fee. Their transaction will complete first, they'll buy whatever you were trying to buy, and your transaction will fail. This is called "front running" and bots are very very good at it.
(In a real exchange or stock market, this is both impractical and illegal. But Eth is play money for techbros who never grew up. We can't expect them to understand modern finance.)
This article is about how an Eth smart contract had a bug that gave out nine million dollars in Eth for basically free, and how the author got the nine million dollars without front running bots stealing it from under him.
well, *actuaalllyyyy*, a functional Bitcoin client need not accept transactions in chronological order. In fact, miners have an incentive to accept the highest fee transaction, regardless of when it was received.
Bitcoin doesn't take transactions in chronological order, you have a deep misunderstanding on how these systems work. If it did 0conf transactions would be safe.
>In a real exchange or stock market, this is both impractical and illegal.
Frontrunning is literally the business model of robinhood. They sell order flow to HFT firms then they execute them as they see fit.
Everything that happens in crypto happens on 'regulated' markets, it's just less obvious to the public.
>Bitcoin doesn't take transactions in chronological order, you have a deep misunderstanding on how these systems work. If it did 0conf transactions would be safe.
I googled, and it looks like you're right. Bitcoin transactions can be frontrun too. I don't know why I expected Bitcoin to be any more secure than any other crypto.
>Frontrunning is literally the business model of robinhood. They sell order flow to HFT firms then they execute them as they see fit.
[Robinhood says they don't:](https://robinhood.com/us/en/support/articles/stocks-order-routing-and-execution-quality/)
>Do you or the market makers use high-speed technology to trade ahead of Robinhood orders?
>No. This practice, known as front-running orders, is illegal.
I mean, you don't have to believe them, but the difference between breaking the law and risking legal liability, and risking nothing because no law can touch anonymous crypto bots, is pretty huge.
For the sake of argument, if Bitcoin miners did strictly add transactions in chronological order it would still be possible to double spend by the sender spending the same coin to two different addresses at (almost) the same time since different miners could get a different transaction first. Both transactions are broadcast through the network, each miner would get one of them with about 50/50 chances while it can show up to the recipient as pending on a explorer.
But even if the first transaction was sent instantaneously to every miner with no chance of double spend or getting ahead in a forced chronological queue a miner could just decide to not include *that one transaction in particular* while choosing to include a later transaction to the same destination from someone cooperating with them.
There is a mention of chronological order of transactions in the whitepaper with blocks:
> In this paper, we propose a solution to the double-spending problem using a peer-to-peer distributed timestamp server to generate computational proof of the chronological order of transactions.
It also used to be possible to send Bitcoins without fees and they would get priority based on how old they were, so that if someone sent sufficiently old coins they would get added and mined for free. So there used to be a time-based priority system of sorts.
The difference between ETH and BTC is mostly that trading can happen directly on the ETH ledger/contracts, the exact trade and planned amounts is announced and executed on ETH, perfect for frontrunning by bots. Bitcoin is just deposited and withdrawn to central exchanges so people only get a vague idea of where Bitcoin is moving to-from with no frontrunning potential on-chain. The frontrunning happens on the central exchanges by the central exchanges.
So moral of the story is unless you're lucky to get a small army of selfless, well connected, dedicated, competent and obviously unpaid volunteers working overtime you can lose millions EVEN IF eventually you notice the fuckup and try to salvage the operation. It's not a dark forest. It's an accursed quagmire.
Not just any dark forest, **the** dark forest.
The author likes a sci-fi book with a environment where being detected or announcing the location of someone else means certain death, which is called "the dark forest".
Which is a good description of this particular situation, a situation that requires a literal hidden conspiracy to avoid detection on a public ledger.
This is really amazing and I respect the people who did this to the highest level. But I am just so shocked after all those agents they interacted with no one just said fuck it and cashed in 10 million. Absolutely amazing and very noble. Glad we have people like this in the space
This is a good read. I have been in this space for what feels like "forever" (2013) but my grasp of certain technical aspects is still rather limited. This was kind of eye opening as for whats happening in the mempool. My questions is: Is this how it was intended? Is this "good for Ethereum"?
As a very general note, I have the feeling that blockchain in general and its value proposition in particular are already very hard to grasp for the general public. But smart contract platforms (not just Ethereum) take it to another level. I have been doing some programming in the past but none of this sort and most of this stuff is way over my head. Clearly this is all in an early stage and the buggy, badly audited smart contracts get weed out quickly (sometimes meaning lots of people lose money), but where is this going in the future, if hardly anyone (any non-programmer) can understand the whole process? Personally, I find DeFi an exciting new application but I am not ready to dump any money into any of those token, no matter what kind of profit other people are making with this. I am more of a conservative crypto guy I suppose, as per Warren: don't invest what you don't understand. I do understand Bitcoin, Ethereum and Doge and I am invested in 2 of those 3.
The network is "working as intended", but I am not sure anyone foresaw that miners might abuse having access to the mempool in such way. I don't think it was intended for people to comb through the mempool to take advantage of it, but surely the possibility was known. Either way, it would be quite difficult to prevent such a thing because -- unless you have a private mining pool -- transactions need to be public.
Maybe zk technologies help here, I don't know.
I'm not a contract developer nor an expert by any means, but from reading this article and the first dark forest article, I took away that the mempool is no different than any mature market, there are ways to get ahead that end up creating changes. In law enforced markets like stocks, it's a law (or a law lobbied for), in the case of the mempool, it seems like a forcing function for correctness. In the first forest article it was a mistake exploited for bad, in this one it was for good.
I can't handle DeFi, it's not for me, but I see it as a stepping stone to maturity. Those fees tho, they definitely are worse for eth than the competition / natural selection going on with the mempool.
That’s smart not to invest in what you don’t understand. That doesn’t mean it won’t see mass adoption. How many people can explain how insurance works or how credit card companies work? Once something is saturated enough in our society and it brings real utility it will gain mass adoption.
Ya, I've never understood this thought--e*specially* when we're talking about something so obscure and specialized as the Ethereum mempool.
I'm a blockchain developer and, even if I know a lot about how Ethereum works, I don't even know much of anything about how MySQL works under-the-hood because, "why do I need to know as long as it works?"
Yet everyone's obsessed with needing to know how blockchain works for them to not think it's terrible. It's the weirdest thing and I have no clue why, with this specific technology, this phenomenon happens. Maybe it's because the technology itself is what's been mostly talked about rather than the products built on it?
Tell me how a credit card transaction works: which company it goes through first, second, third, who profits where, whose reserves money sits in for how long until they transfer it to the next company, how they make that transfer, where it ends up, who's responsible at every step of the process, where's the cash (is there any?) etc. Oh ya, and all the software underlying it all because that's the level of detail we're going through with blockchain.
I'll then tell you how a blockchain transaction works: I send a transaction (swipe a card), a miner makes sure my account has enough balance according to their own records then sends it to other miners to check, then they all agree to change the balance in my account.
Then we can talk about which one is more complicated.
Firstly, they don't have to do anything. They are choosing to.
Secondly, unless we begin letting AI code software for us. There will always be bugs in everything including smart contracts. One of my family members once had $70,000 deposited into his bank account from a banking error. I know 70k isn't 10 million but there's bugs everywhere.
... and the bank undid that deposit when it was discovered. I don't even have to ask you. That reversal was part of an *intentionally adopted process* for dealing with *expected errors*.
Ethereum was sold on the promise of "code is law". Everything was supposed to be automatic, and therefore cheaper, fairer, harder to tamper with, safer, and more predictable than the old "manual" processes.
This of course was premised on the foolish assumption that there would be no bugs. No steps were taken to *assure* that there would *be* no bugs; in fact, the programming language provided was a giant mine field. No defined mechanisms were provided for *dealing* with bugs. No expectations were set about what would even be *considered* a bug.
Right out of the gate, people created "the DAO". It had a stupid business model. It also had a stupid bug in it, and, in spite of its having been blessed by the leading lights of the Ethereum community, somebody immediately drained it of something like 80 million US dollars. The response was to *fork the entire blockchain* to add a special case... violating a whole bunch of direct promises in the process.
Then there've been a long series of smaller losses, ending up in this "dark forest" situation. And, I'd like to point out, the system has moved money around, but created very little real tangible value, as opposed to enabling weird speculative financial instruments. You may value its *intentional* redistribution of value (I don't)... but most likely Ethereum's *unintentional* redistribution via bugs has exceeded all the *other* value it's created.
And now we have people hacking around the system to rescue others from their stupidity. No, they don't "have to" do that in the sense that anybody is forcing them to do it. Yes, they *do* "have to" do it in the sense that **the system provides no other way to achieve their desired outcome**. And their desired outcome is in fact what the system was **sold as providing**.
That isn't cheaper than the old manual/legal processes; it involves a lot of skilled work. It's not fairer or harder to tamper with than the old processes; it puts everybody's assets at the mercy of whoever can find the most bugs. It's not safer or more predictable than the old processes; you may or may not be able to deal with any given fuckup.
Yes, (nearly) everything has bugs. A smart person designs a system that takes that into account from the beginning. An idiot assumes that everything will work perfectly, builds a giant Rube Goldberg machine, makes a bunch of "code is law" promises, and responds to any undesired consequences by weird ad-hoc patches or by trying to find "bugs in the bugs".
Until such time as these failures can be made *vanishingly rare*, smart contracts ADD NO VALUE WHATSOEVER. If that's even possible, you could even make a good argument that every time somebody is rescued from a screwup, it reduces the incentives to actually make the system work. It *definitely* distorts people's incentives to reduce risk by *not using the broken system*. So maybe the most "helpful" thing to do would be just to let idiots take their losses.
I still don't understand what's happening at the core of this and the other dark forest post from a few weeks ago. How exactly are these bots front-running/stealing the ethereums?
-these bots scan the smart contracts that are waiting to be executed by the miners
-the bots find vulnerabilities (another grey area in my mind) in the contract
-the bots adjust the destination address of where the contract is supposed to send the the ethereums
-then the bots continually execute the vulnerable smart contract code
All this research into smart contracts and crytpocurrency may seem pointless and a waste of time. It is very risky to dabble in, and I don't think assigning value to these "bitcoins," or whatever they may be called, will be the lasting effect of all this research. Perhaps some new programming language, or something we haven't even thought of, could be the result of these people working on the outer edges of current knowledge.
I love that they're continuing the Dark Forest analogy! Makes me also realize I never want to dip my toe in crypto like that. It's like an amateur going up to an entirely unregulated wall street and expecting to earn some quick cash.
I quickly want to point out that we've recently seen a surge in uniswap/bancor based "liquidity pools" (all projects copying each other). The main idea here is that you can lock up your crypto in a smart contract - which is considered "secure" as to no one can steal it (audited code by reputable companies and such). If true the risk is very small with things like impermanent loss, which doesn't apply to all pools.
The idea here is that your money is provided liquidity and you'll get paid a portion of the fees as well as some new token which can have a very high value (for a fleeting moment).
This is important to realize when looking at the crazy marketing around these projects, if it's based on uniswap you can reasonably sure your principal won't get stolen - regardless of the scammy and weird marketing.
interesting read - seems like the solution to the dark forest is equivalent to a dark pool in traditional finance?
the logical conclusion is that within a few months we'll have dark pools run by miners who will process your transactions without broadcasting to mempool, in exchange for an increased gas fee. and, within a year, we'll find out that some dark pools sold order flow to those HFT's anyways, a la UBS https://sites.law.berkeley.edu/thenetwork/2015/01/29/ubs-dar...
i tried writing some toy Ethereum smart contracts circa 2016. at that time it was immensely difficult to write them in a secure way -- even a simple "hello world" level Solidity contract could easily have exploitable bugs if you don't code in an extremely defensive style.
i'm told things have improved since then -- can anyone who's used Solidity more recently comment on this? is it true?
this, plus the fact that putting information from the real world onto the blockchain unavoidably requires some trust, seemed like the two big problems then, and it seems like they haven't really been fixed.
Love whitehat crypto postmortems like this. They always read like heist movies.
Curious about the use of SparkPool to bypass the mempool and get the transactions minted directly into a block. It looks like anyone can sign up and contribute their hashrate to SparkPool. Is there a risk of malicious miners running workers in their competitors' pools and then frontrunning?
Makes me think of salvage operations, and then raises the question of how do people get paid? They're providing a valuable service. I think in shipping there are both conventions and an ability to quickly negotiate that allows contracting for a salvage ship to rush to the aid of a grounded or sinking container vessel.
cryptocurreny != investment scam.
It's just another way to transfer and store value.
Interacting with automated contracts is an interesting extension to that system which can make things alot more complex.
The 'dark forest' comes from a kind of man-in-the-middle attack where anyone can see the order book and exploit it, by putting their own slightly better orders in. Hence the need for co-operation with a closed order book (miner) to get the transaction in safely.
I offer that anyone who did the work that these researchers did would have also been “rightful owners” of that money.
This is the consequence of programmable money; there’s no getting around it, and, in my opinion, people shouldn’t want to. Rescuing people and brands who don’t put the effort into security from the consequences of their own mistakes isn’t a net benefit.
I'm all for anonymous teams, but look at the hoops this person had to jump through just to get in touch with them to report the bug.
When you're anonymous, all you have is your brand, and theirs should have burned to the ground for this entirely preventable error.
This is all very interesting to read about, but in the same way epic battles in Eve Online are interesting to read about but not participate in. I hope the author doesn't think this article is functioning as an enticement to use ETH myself, because it's only confirming for me that I never, ever want any of my money near that shambling wreck.