feat: estimate gas premium from mempool

[wjmelements]

Reviewer @rvagg
This premium recommendation change does not require FIP-0115 and can be released early.
I'm still running simulations to see if the system can be improved.
The gas market simulation covers:

• suggesting premium for incoming transactions
• message selection
• base fee adjustment

Changes

• GasEstimateGasPremiumFromMempool
• tests
• gas market simulation tool

Simulation

I built a tool for simulating the gas market.

Method

The simulation runs over a configurable number of epochs.
There are a configurable number of incoming transactions per epoch.
They have a distribution of possible gas limits and deadlines.
They query GasEstimateGasPremium and then GasEstimateFeeCap and then join the mempool.
The epoch concludes with tq-proposer message selection.
The metrics track how many transactions met their deadlines with a breakdown by block.

Results

I ran the simulation over 2000 epochs with a starting base fee of 1000.
Briefly, an explanation of the keys in the tables:

• gpe: average gas per epoch, as a multiple of the block gas limit; also 1/10 of the number of submitted transactions per epoch
• baseFee@2000: the base fee at the end of the simulation
• delayed: how many transactions were late and not yet confirmed at the end of the simulation
• ever-late: How many transactions were ever late, broken down by their nblocksincl (nb)
• gas%: average gaslimit utilization as a percentage of 5 proposer blocks

I also ran for gpe as high as 5 but it seems impossible to digest much faster than 2.8.

TQ proposals seem to have a maximum average digestion rate between 2.8 and 3.0 due to nearby TQs.

GasEstimateGasPremiumFromMempool

  ┌─────┬──────────────┬─────────────────────┬──────────────────┬───────┬───────┬───────┬───────┐
  │ gpe │ baseFee@2000 │ delayed (nb1/2/10)  │    ever-late     │ nb1%  │ nb2%  │ nb10% │ gas%  │
  ├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
  │ 2.2 │ 115          │ 4 (2/2/0)           │ 747/44K (1.7%)   │ 9.6%  │ 2.8%  │ 0.2%  │ 43.9% │
  ├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
  │ 2.4 │ 119          │ 6 (3/1/2)           │ 1100/48K (2.3%)  │ 12.0% │ 3.7%  │ 0.5%  │ 47.9% │
  ├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
  │ 2.6 │ 191          │ 7 (2/0/5)           │ 1598/52K (3.1%)  │ 14.4% │ 4.9%  │ 1.0%  │ 52.0% │
  ├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
  │ 2.8 │ 943          │ 42 (5/12/25)        │ 2132/56K (3.8%)  │ 14.0% │ 5.8%  │ 1.8%  │ 55.9% │
  ├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
  │ 3.0 │ 8905         │ 1776 (101/261/1414) │ 4188/60K (7.0%)  │ 18.6% │ 8.6%  │ 4.8%  │ 57.7% │
  └─────┴──────────────┴─────────────────────┴──────────────────┴───────┴───────┴───────┴───────┘

Simple (BaseFee / 8 + noise)

Tool option --flat-premium

┌─────┬──────────────┬─────────────────────┬──────────────────┬───────┬───────┬───────┬───────┐
│ gpe │ baseFee@2000 │ delayed (nb1/2/10)  │    ever-late     │ nb1%  │ nb2%  │ nb10% │ gas%  │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 2.2 │ 937          │ 0 (0/0/0)           │ 768/44K (1.7%)   │ 13.3% │ 2.3%  │ 0.0%  │ 43.6% │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 2.4 │ 953          │ 0 (0/0/0)           │ 1013/48K (2.1%)  │ 15.2% │ 3.1%  │ 0.0%  │ 48.0% │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 2.6 │ 993          │ 0 (0/0/0)           │ 1563/52K (3.0%)  │ 17.7% │ 5.5%  │ 0.2%  │ 51.7% │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 2.8 │ 1001         │ 12 (3/6/3)          │ 3376/56K (6.0%)  │ 27.1% │ 11.0% │ 1.7%  │ 56.3% │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 3.0 │ 1001         │ 2169 (225/461/1483) │ 6717/60K (11.2%) │ 28.4% │ 14.8% │ 7.7%  │ 57.3% │
└─────┴──────────────┴─────────────────────┴──────────────────┴───────┴───────┴───────┴───────┘

Historical

Tool option --legacy-premium
With 100k premium floor (150k for nb2, 200k for nb1)

┌─────┬──────────────┬─────────────────────┬──────────────────┬───────┬───────┬───────┬───────┐
│ gpe │ baseFee@2000 │ delayed (nb1/2/10)  │    ever-late     │ nb1%  │ nb2%  │ nb10% │ gas%  │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 2.2 │ 817134       │ 1 (1/0/0)           │ 490/44K (1.1%)   │ 8.3%  │ 1.5%  │ 0.0%  │ 43.6% │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 2.4 │ 844674       │ 0 (0/0/0)           │ 821/48K (1.7%)   │ 10.7% │ 3.2%  │ 0.0%  │ 48.0% │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 2.6 │ 852502       │ 2 (0/2/0)           │ 1497/52K (2.9%)  │ 16.1% │ 5.5%  │ 0.3%  │ 51.7% │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 2.8 │ 883219       │ 39 (16/14/9)        │ 3077/56K (5.5%)  │ 24.2% │ 9.1%  │ 1.9%  │ 56.3% │
├─────┼──────────────┼─────────────────────┼──────────────────┼───────┼───────┼───────┼───────┤
│ 3.0 │ 955230       │ 1867 (166/337/1364) │ 5345/60K (8.9%)  │ 25.7% │ 12.6% │ 5.5%  │ 57.4% │
└─────┴──────────────┴─────────────────────┴──────────────────┴───────┴───────┴───────┴───────┘

[github-actions]

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

[github-actions]

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

PR title now matches the required format.

[wjmelements]

force pushed a rebase onto #13531

[LinZexiao]

This simulation tool provides an intuitive evaluation of the FIP-0115 algorithm, and the chosen metrics effectively capture the trade-off between network resource utilization and user transaction costs. To further refine the evaluation, I suggest the following improvements:

1. Scenario Expansion: Simulating Traffic Downswings and Volatility

The current simulations primarily focus on performance under static message pressure. I recommend introducing a dynamic traffic model to observe the algorithm’s behavior when message pressure suddenly drops (downswing) or fluctuates significantly.

• Objective: To verify if the algorithm's Premium suggestions lag due to "mempool inertia," which could trigger unnecessary oscillations in the $BaseFee$.

2. Enhanced Metrics: Volatility and Cumulative Expenditure

I suggest adding the following two core metrics to the simulation output:

• BaseFee Volatility: Measures the standard deviation or period-over-period change rate of the $BaseFee$. Lower volatility indicates better price predictability for users.
• Total Cumulative Expenditure: The aggregate sum of costs for all included messages ( $\sum GasUsed \times GasPrice$ ) across the entire simulation period. While a snapshot of the $BaseFee$ (e.g., at epoch 2000) might be skewed by transient fluctuations, cumulative throughput provides a more robust measure of how efficiently the algorithm utilizes network capacity over a full cycle.

[wjmelements]

Total Cumulative Gas Throughput

This is already measured, and is reflected as the percentage gas% in the result column as a percentage of maximum theoretical throughput.

It seems to cap out around 57.5%, perhaps because of inherent TQ overlap.

[LinZexiao]

Total Cumulative Gas Throughput累计总天然气吞吐量

This is already measured, and is reflected as the percentage gas% in the result column as a percentage of maximum theoretical throughput.这已经测量过了，并在结果列中以 gas% 占最大理论吞吐量的百分比）的形式反映出来。

It seems to cap out around 57.5%, perhaps because of inherent TQ overlap.似乎最高值在 57.5% 左右，可能是由于固有的 TQ 重叠造成的。

I apologize, my previous state was incorrect, and I have now corrected it. I think the Average Gas Price effectively reflects the actual amount of tokens users spend to get their messages included on-chain.

[wjmelements]

Total Cumulative Expenditure

This is a good idea. Can also track how much FIL was burned vs how much was paid to proposers. (assumes gaslimit = gasUsed)

[LinZexiao]

Additionally, some extreme scenarios may be taken into consideration, such as a surge of large messages (e.g., SettleDealPaymentsExported). To account for this, we could shift the mean simulated message gas limit from 10% to 75% of the block limit.

[Kubuxu]

Seems to me a bit overly complex, but just a bit. I'm glad we are not running an actual selection process to move forward an epoch, because that could be cost-prohibitive.

Have you tested the results on a live mainnet node for comparison with the old method?
I need a bit more time to think.

[wjmelements]

I noticed some irregularities in the results of eth_maxPriorityFeePerGas this weekend. I will investigate them tomorrow.

[wjmelements]

I noticed some irregularities in the results of eth_maxPriorityFeePerGas this weekend. I will investigate them tomorrow.

I think the behavior is correct, but during the investigation I identified some ways to improve:

• target inclusion in a non-greedy block (previously suggested by @LinZexiao)
• measure total gas premiums paid in the simulation
• Disregard mempool transactions that are blocked from inclusion because of a nonce gap

[Kubuxu]

However I still find the evidence from the simulation more compelling

Of course, testing it in live is more of a smoke test to check if there are no unexpected factors.

[wjmelements]

Per a request from @magik6k I have collected some tx history data. Here it is:

The current gas market simulation generates more midsized transactions, while the actual distribution has almost exclusively tiny transactions.

I will change the incoming transaction modeling so that it matches this observed distribution.

[wjmelements]

Next I am working on the requested "replay" simulation. In order for this to be fair I would need to record actual mempool arrivals. I'll set that up today and run it for at least 24 hours.

[wjmelements]

Next I am working on the requested "replay" simulation. In order for this to be fair I would need to record actual mempool arrivals. I'll set that up today and run it for at least 24 hours.

It took longer to collect data than expected due to two disruptive major internet outages yesterday and the day before. I have a 24 hour period recorded now though so I'll be doing the replay analysis today.

[LinZexiao]

Is this PR going to be merged before time for NV28 calibnet ( 2026-04-27 , according to upgrading plan) ?

[wjmelements]

Is this PR going to be merged before time for NV28 calibnet ( 2026-04-27 , according to upgrading plan) ?

I haven't had time yet to work through the last few issues, but this doesn't block nv28.

[wjmelements]

Disregard mempool transactions that are blocked from inclusion because of a nonce gap

I'm currently optimizing for this on a separate branch in order to avoid doing GetActorAfter for every sender. While there is already a nonce cache, my plan is to put startNonce and startBalance into msgSet. Such an optimization should already benefit message validation (Add) and message selection, so I plan to merge it separately and ahead of this change.