holiman

holiman

Member Since 12 years ago

Stockholm, Sweden

Experience Points
471
follower
Lessons Completed
2
follow
Lessons Completed
6
stars
Best Reply Awards
87
repos

1040 contributions in the last year

Pinned
⚡ Fixed size 256-bit math library
⚡ Evm laboratory
⚡ Implementation of the ciphers in iClass
⚡ vmstats
⚡ Face-meltingly fast, thread-safe, marshalable, unionable, probability- and optimal-size-calculating Bloom filter in go
⚡ Official Go implementation of the Ethereum protocol
Activity
Jan
23
1 day ago
push

holiman push holiman/go-ethereum

holiman
holiman

eth/protocols/snap: make snap look up code by prefix only

commit sha: ea408fa440c922699a774c18ec43fae87519e9cc

push time in 9 hours ago
push

holiman push holiman/go-ethereum

holiman
holiman

cmd/devp2p: more work on snap protocol tests

commit sha: 6d07a34a47b31a14d2720b45b3ac992ffe219134

push time in 10 hours ago
push

holiman push holiman/go-ethereum

holiman
holiman

cmd/devp2p/internal: minor test fixes

holiman
holiman

eth/protocols/snap: make snap return data even if bytes=0

commit sha: 1321db516d861775d8acff60c75334d1cbca2e18

push time in 11 hours ago
push

holiman push holiman/go-ethereum

holiman
holiman

cmd/devp2p: run tests as regular tests + implement proof/trie verification

commit sha: 7087a1cfb7aa5e7a6da4ef0a36911d448716629c

push time in 11 hours ago
pull request

holiman pull request ethereum/go-ethereum

holiman
holiman

cmd/devp2p: implement snap protocol testing

This is a WIP

This PR adds support in the devp2p binary to perform testing of the snap protocol.

  • GetAccountRange
  • GetStorageRange
  • GetByteCodes
  • GetTrieNodes

Later on, the idea is to use this functionality for Hive-testing.

Testing locally

In case other client implementors want to try out the testcases, here are two scripts that can be used for running it.

A script to set up the Node Under Test (customize this for your node):

$ cat setup.sh 
geth=~/go/src/github.com/ethereum/go-ethereum/build/bin/geth
ddir=./foo
genesis=~/go/src/github.com/ethereum/go-ethereum/cmd/devp2p/internal/ethtest/testdata/genesis.json
chain=~/go/src/github.com/ethereum/go-ethereum/cmd/devp2p/internal/ethtest/testdata/halfchain.rlp

set -e

rm -rf $ddir
# init
$geth --datadir $ddir --nodiscover --nat=none --networkid 19763 --verbosity 5 init $genesis
# import 
$geth --datadir $ddir --nodiscover --nat=none --networkid 19763 --verbosity 5 import $chain
# run
$geth --datadir $ddir --nodiscover --nat=none --networkid 19763 --verbosity 5 --nodekeyhex 1fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff console

And this is how I run the tests:

$ cat testit.sh 
devp2p=~/go/src/github.com/ethereum/go-ethereum/devp2p
genesis=~/go/src/github.com/ethereum/go-ethereum/cmd/devp2p/internal/ethtest/testdata/genesis.json
chain=~/go/src/github.com/ethereum/go-ethereum/cmd/devp2p/internal/ethtest/testdata/halfchain.rlp


enode="enode://a6c36f0904a863815b366120df8536f0c1477d9213fcc2421a0efa4eae64abf83[email protected]127.0.0.1:30303"
$devp2p rlpx snap-test $enode $chain $genesis
pull request

holiman pull request ethereum/go-ethereum

holiman
holiman

cmd/devp2p: implement snap protocol testing

pull request

holiman pull request ethereum/go-ethereum

holiman
holiman

cmd/devp2p: implement snap protocol testing

push

holiman push holiman/go-ethereum

holiman
holiman

cmd/devp2p: implement snap protocol testing

commit sha: 70b45a14f53203ba4f1347ad90671ea5bd1979ed

push time in 15 hours ago
push

holiman push holiman/go-ethereum

holiman
holiman

cmd/devp2p/internal/ethtest: work on snap tests

commit sha: bd017c703b86202a572953be4c510726c8b656fe

push time in 16 hours ago
Jan
22
2 days ago
Activity icon
issue

holiman issue comment ethereum/execution-apis

holiman
holiman

Engine API: proposal for authentication

This is an RFC for authenticated JSON rpc API

Authentication

The engine JSON-RPC interface, exposed by EL and consumed by CL, needs to be authenticated. The authentication scheme chosen for thus purpose is JWT.

The type of attacks that this authentication scheme attempts to protect against are the following:

  • RPC port exposed towards the internet, allowing attackers to exchange messages with EL engine api.
  • RPC port exposed towards the browser, allowing malicious webpages to submit messages to the EL engine api.

The authentication scheme is not designed to

  • Prevent attackers with capability to read ('sniff') network traffic from reading the traffic,
  • Prevent attackers with capability to read ('sniff') network traffic from performing replay-attacks of earlier messages.

Authentication is performed as follows:

  • For HTTP dialogue, each jsonrpc request is individually authenticated by supplying JWT token in the HTTP header.
  • For a WebSocket dialogue, only the initial handshake is authenticated, after which the message dialogue proceeds without further use of JWT.
  • For inproc, a.k.a raw ipc communication, no authentication is required, under the assumption that a process able to access ipc channels for the process, which usually means local file access, is already sufficiently permissioned that further authentication requirements do not add security.

JWT specifications

  • The EL MUST support the at least the following alg HMAC + SHA256 (HS256)
  • The EL MUST reject the alg none.

The HMAC algorithm means that symmetric encryption is used, thus several CL's will be able to use the same key, and, from an authentication perspective be able to impersonate each other. From a deployment perspective, it means that an EL does not need to be provisioned with individual keys for each caller.

Key distribution

The EL and CL clients MUST accept a cli/config parameter: jwt-secret, a 256 bit key, to be used for verifying/generating jwt tokens. If such a parameter is not given, the client SHOULD generate such a token, valid for the duration of the execution, and show the token in the output, which the user can then use to provision the counterpart client with.

holiman
holiman

Yup. I can get started tomorrow at the earliest, if anyone wants to get it going before that please go ahead 

Activity icon
created branch

holiman in holiman/go-ethereum create branch devp2p-snap

createdAt 1 day ago
Activity icon
issue

holiman issue comment bloXroute-Labs/go-ethereum

holiman
holiman

Private Transaction API Sample (v1.10.13)

holiman
holiman

Actually, thinking about it some more. that whole thing I wrote about pruning locking up is inaccurate. ts.After(now) is always false for ts==0, so the loop will continue, thus prune the item. 

Jan
21
3 days ago
Activity icon
issue

holiman issue comment bloXroute-Labs/go-ethereum

holiman
holiman

Private Transaction API Sample (v1.10.13)

holiman
holiman

Thanks for double-checking :)Yes, I'm aware. A flashbot member pointed me to take a look here, and see if I saw any problems with this feature, so I did. 

open pull request

holiman wants to merge ethereum/go-ethereum

holiman
holiman

eth/tracers: native prestate tracer

Some data points:

  • Tx 0x90b4a6727f8ca999feb22578335b6447decf4cf559724979dd67aaf9c59beda2: 0m0.454s vs 0m0.044s
  • Tx 0xb5b072dcff92fca42ef863052ee4eecaed2a90c0c769273500180afe277ad086: 0m1.005s vs 0m0.065s
  • Tx 0xfd16a9f644c22344fa44ec7aaa600d078356e3d1632bf47c2bb490534f7d7f97: 0m0.344s vs 0m0.032s
  • Tx 0xfb23d3cff44815fc4da75b04446bc2a04afbc900a008a3cd0b322a497e028b5f: 0m0.185s vs 0m0.028s
holiman
holiman

Maybe switch here instead?

pull request

holiman merge to ethereum/go-ethereum

holiman
holiman

eth/tracers: native prestate tracer

Some data points:

  • Tx 0x90b4a6727f8ca999feb22578335b6447decf4cf559724979dd67aaf9c59beda2: 0m0.454s vs 0m0.044s
  • Tx 0xb5b072dcff92fca42ef863052ee4eecaed2a90c0c769273500180afe277ad086: 0m1.005s vs 0m0.065s
  • Tx 0xfd16a9f644c22344fa44ec7aaa600d078356e3d1632bf47c2bb490534f7d7f97: 0m0.344s vs 0m0.032s
  • Tx 0xfb23d3cff44815fc4da75b04446bc2a04afbc900a008a3cd0b322a497e028b5f: 0m0.185s vs 0m0.028s
Activity icon
issue

holiman issue comment ethereum/execution-apis

holiman
holiman

Engine API: proposal for authentication

This is an RFC for authenticated JSON rpc API

Authentication

The engine JSON-RPC interface, exposed by EL and consumed by CL, needs to be authenticated. The authentication scheme chosen for thus purpose is JWT.

The type of attacks that this authentication scheme attempts to protect against are the following:

  • RPC port exposed towards the internet, allowing attackers to exchange messages with EL engine api.
  • RPC port exposed towards the browser, allowing malicious webpages to submit messages to the EL engine api.

The authentication scheme is not designed to

  • Prevent attackers with capability to read ('sniff') network traffic from reading the traffic,
  • Prevent attackers with capability to read ('sniff') network traffic from performing replay-attacks of earlier messages.

Authentication is performed as follows:

  • For HTTP dialogue, each jsonrpc request is individually authenticated by supplying JWT token in the HTTP header.
  • For a WebSocket dialogue, only the initial handshake is authenticated, after which the message dialogue proceeds without further use of JWT.
  • For inproc, a.k.a raw ipc communication, no authentication is required, under the assumption that a process able to access ipc channels for the process, which usually means local file access, is already sufficiently permissioned that further authentication requirements do not add security.

JWT specifications

  • The EL MUST support the at least the following alg HMAC + SHA256 (HS256)
  • The EL MUST reject the alg none.

The HMAC algorithm means that symmetric encryption is used, thus several CL's will be able to use the same key, and, from an authentication perspective be able to impersonate each other. From a deployment perspective, it means that an EL does not need to be provisioned with individual keys for each caller.

Key distribution

The EL and CL clients MUST accept a cli/config parameter: jwt-secret, a 256 bit key, to be used for verifying/generating jwt tokens. If such a parameter is not given, the client SHOULD generate such a token, valid for the duration of the execution, and show the token in the output, which the user can then use to provision the counterpart client with.

holiman
holiman

Is there any limits to what is a valid jwt-secret, or is any 256-bit value acceptable?

Well, we can prohibit 0x00, but not sure what else we can do.

Is there any expectation to include exp (expiration) on the tokens

The idea with iat would be to make it so that there's a 5 (?) - second window where it is valid. So it would implicitly carry the exp. Or rather, the expiry would be enforced on the EL, not settable by the CL.

The canonical usecase for exp is when a federated server issues "this is John, he's an admin, token is valid for one hour". It's not quite our usecase

pull request

holiman merge to ethereum/execution-apis

holiman
holiman

Engine API: extend semantics of `executePayload` and `forkchoiceUpdated` methods

The following changes to executePayload and forkchoiceUpdated methods are proposed:

  • Makes execution semantics optional for executePayload, and renames this method to newPayload to make the name sound
  • Adds an option to execute payload on receiving forkchoiceUpdated before updating the forkchoice state
  • Allows for EL client to skip updating the forkchoice state if forkchoiceUpdated references a historical block from the canonical chain that has been validating a while ago, i.e. forkchoice state may never go backwards if an update to the same chain is requested
  • Introduces ACCEPTED status. It may be sent in the response in the following cases:
    • If CL sends a payload from a side fork and the parent block is missing, EL client software may just store this payload locally and no sync or payload validation process won't be initiated until this payload (or its descendant) will be designated as the head of the canonical chain
    • If CL sends forkchoiceUpdated to make set the head to the payload that does exist and can be validated (parent block and state are available locally), but validation is not yet done and client software implementation doesn't induce validation process on forkchoiceUpdated. Probably, in this case it is better for EL client software to respond with IGNORED to better reflect the semantics of this response
  • Requires payload execution on the canonical chain to be unaffected by an active sync process if it is happening on a side branch of the block tree, and data required to validate the payload (the parent block and state) is locally available. This requirement disallows EL client software to respond SYNCING if newPayload sends a child of the head of the canonical chain when the software is catching up with the head of a side fork, it prevents node to turn optimistic (as per optimistic sync spec) for no reason

Additionally:

  • Moves Payload validation process to a separate section for the sake of modularity.
  • Introduces Sync process section to give a notion of the sync process to the spec. It may be removed if deemed superfluous, but could be useful in some sense

A separate EXECUTING/VALIDATING status has been previously considered to denote the state of EL client software in which it executes multiple blocks in a row in order to obtain the parent state to be able to validate a payload. But this operation has been kept under SYNCING status with the reflection in the Sync process description. Distinguishing these two different states is not that useful from the CL client software perspective as both doesn't give an understanding of how much time the operation will take to inform the CL behaviour.

open pull request

holiman wants to merge ethereum/execution-apis

holiman
holiman

Engine API: extend semantics of `executePayload` and `forkchoiceUpdated` methods

The following changes to executePayload and forkchoiceUpdated methods are proposed:

  • Makes execution semantics optional for executePayload, and renames this method to newPayload to make the name sound
  • Adds an option to execute payload on receiving forkchoiceUpdated before updating the forkchoice state
  • Allows for EL client to skip updating the forkchoice state if forkchoiceUpdated references a historical block from the canonical chain that has been validating a while ago, i.e. forkchoice state may never go backwards if an update to the same chain is requested
  • Introduces ACCEPTED status. It may be sent in the response in the following cases:
    • If CL sends a payload from a side fork and the parent block is missing, EL client software may just store this payload locally and no sync or payload validation process won't be initiated until this payload (or its descendant) will be designated as the head of the canonical chain
    • If CL sends forkchoiceUpdated to make set the head to the payload that does exist and can be validated (parent block and state are available locally), but validation is not yet done and client software implementation doesn't induce validation process on forkchoiceUpdated. Probably, in this case it is better for EL client software to respond with IGNORED to better reflect the semantics of this response
  • Requires payload execution on the canonical chain to be unaffected by an active sync process if it is happening on a side branch of the block tree, and data required to validate the payload (the parent block and state) is locally available. This requirement disallows EL client software to respond SYNCING if newPayload sends a child of the head of the canonical chain when the software is catching up with the head of a side fork, it prevents node to turn optimistic (as per optimistic sync spec) for no reason

Additionally:

  • Moves Payload validation process to a separate section for the sake of modularity.
  • Introduces Sync process section to give a notion of the sync process to the spec. It may be removed if deemed superfluous, but could be useful in some sense

A separate EXECUTING/VALIDATING status has been previously considered to denote the state of EL client software in which it executes multiple blocks in a row in order to obtain the parent state to be able to validate a payload. But this operation has been kept under SYNCING status with the reflection in the Sync process description. Distinguishing these two different states is not that useful from the CL client software perspective as both doesn't give an understanding of how much time the operation will take to inform the CL behaviour.

holiman
holiman

From acd today: this should be made stricter, to not make it implementation-dependent so that clients all lazily choose not to validate any newPayloads at all

Activity icon
issue

holiman issue comment ethereum/pm

holiman
holiman

Ethereum Core Devs Meeting 130 Agenda

Meeting Info

Agenda

holiman
holiman

Last week, we briefly touched upon a slew of 'potentially shanghai eips', one of them EIP-4444, which I know a lot of people are optimistic about and want to push forward, since it's something of a prerequisite for 4488.

However, 4444 says this, about where to get blocks from>

For the purposes of this proposal, we assume clients always start with a configured and valid weak subjectivity checkpoint. The way this is achieved is out-of-scope for this proposal.

And also

We suggest that a specialized “full sync” client is built.

I think it's fine if it's left out of 4444, but in practice, IMO these things needs to be discussed and ironed out and put into production before we go ahead with 4444. And these are two distinct problems: obtaining WSC versus obtaining all blocks. Who put what where, and why (any incentives?), and how is the data discovered and downloaded?

If we have any time over, maybe this could be brought up?

Activity icon
issue

holiman issue comment ethereum/go-ethereum

holiman
holiman

core/rawdb: simple legacy receipt converter

This is a slower but simpler version of https://github.com/ethereum/go-ethereum/pull/23454. I still plan to add in batch reading.

holiman
holiman

From discussion last week: the check that is performed on every node at startup, iterating through the first 40K blocks (on mainnet) to find the first receipt and check legacy, should be improved (since it affects the boot phase of all users). For this check, it's sufficient to look at the indices-structure and figure out the first non-empty block, without actually loading the data.

Otherwise this LGTM. @fjl did you have anything to add -- I left that call early ?

push

holiman push ethereum/go-ethereum

holiman
holiman

trie: test for edgecase in VerifyRangeProof (#24257)

  • trie/proof: edge case for VerifyRangeProof

  • more consistency with other tests in the file

  • trie: fix test todo

Co-authored-by: Martin Holst Swende [email protected]

commit sha: 2dfa4bcf6cb5263b8509722ffd14ddd02eddf47a

push time in 2 days ago
pull request

holiman pull request ethereum/go-ethereum

holiman
holiman

trie: test for edgecase in VerifyRangeProof

This is probably a case where I am setting up something incorrectly, though the test fails when I remove the 1st elements of keys and vals (but passes otherwise). I was hoping you could take a look, since I couldn't exactly figure it out.

I'm assuming it's because there's a shared prefix on the keys, though not sure.

This is the stack trace I am getting:

panic: trie.hashNode: invalid node: <a94e2bd8ee1e6b49b14831e8bd8a7fa01f4582bd1c22e514468c62ee05ca144e>

goroutine 19 [running]:
testing.tRunner.func1.2({0x43a35a0, 0xc000212620})
	/usr/local/Cellar/go/1.17.2/libexec/src/testing/testing.go:1209 +0x36c
testing.tRunner.func1()
	/usr/local/Cellar/go/1.17.2/libexec/src/testing/testing.go:1212 +0x3b6
panic({0x43a35a0, 0xc000212620})
	/usr/local/Cellar/go/1.17.2/libexec/src/runtime/panic.go:1047 +0x266
github.com/ethereum/go-ethereum/trie.hasRightElement({0x445dc38, 0xc00018ea00}, {0xc0001a2ac0, 0x20, 0x40})
	/.../go-ethereum/trie/proof.go:431 +0x579
github.com/ethereum/go-ethereum/trie.VerifyRangeProof({0x60, 0x78, 0xfa, 0xa7, 0x31, 0x9c, 0x98, 0x7c, 0x67, 0xe5, ...}, ...)
	/.../go-ethereum/trie/proof.go:566 +0x10eb
open pull request

holiman wants to merge bloXroute-Labs/go-ethereum

holiman
holiman

Private Transaction API Sample (v1.10.13)

holiman
holiman

Looks like anything added as a private transaction will always make the hash linger for three days, even if the tx was mined two days ago. Maybe consider pruning this on new blocks too?

open pull request

holiman wants to merge bloXroute-Labs/go-ethereum

holiman
holiman

Private Transaction API Sample (v1.10.13)

holiman
holiman

Is this really needed? As long as the private ones are synced with a peer, isn't that enough? This feed might be used by other subsystems, do you really need to prevent them going out there too? Just checking

open pull request

holiman wants to merge bloXroute-Labs/go-ethereum

holiman
holiman

Private Transaction API Sample (v1.10.13)

holiman
holiman

I think there's a possible memleak here. If the same tx is added twice (either it is present twice in the news, or it's added a second time. Then it will be added to the internal list twice. So list is [ txhash, txhash, ...]. timestamps are { txhash: last_timestamp, ..}. After pruning, the first txhash will be removed, and the timestamp is removed.

Next time pruning happens, the timestamp lookup will return 0, meaning that no more pruning will ever occur. If there are other txs that slipped in in-between, the 0-timestamp-element will sit "unsorted", and eventually become the first element, causing pruning to halt.

This can be made self-healing, if instead of storing timestamp of "when to delete", you change it to "when was it added". Because then, if there's a mismatch and the timestamp is missing, the value 0 will cause it to be pruned.

Previous