Problem Statement

If we make VirtualFile write path truly async using tokio-epoll-uring, should we then remove the spawn_blocking + Handle::block_on usage upstack in the same commit?

No, because if we’re still using std-fs , we’d then block the executor in those places where previously we were protecting us from that through the spawn_blocking .

So, if we want to see benefits from tokio-epoll-uring on the write path while also preserving the ability to switch between tokio-epoll-uring and std-fs , where std-fs will behave identical to what we have now, we need to conditionally use spawn_blocking + Handle::block_on

I.e., in the places where we use that know, we’ll need to make that conditional based on the currently configured io engine.

Assessing The Code Base

Who uses the VirtualFile write path?

create_image_layers: doesn’t use spawn_blocking, uses
- ImageLayerWriter ⇒ … ⇒ VirtualFIle::write_*
- par_fsync_async
flush_frozen_layer
- repartitioning happens async already
- create_delta_layer: is an async fn but it uses one spawn_blocking for all it does, which is:
  - async fn frozen_layer.write_to_disk (block_on)
    - the reads from the EphemeralFile are already async
    - but the put_value_bytes writer is sync
  - par_fsync
compact_level0_phase1
- doesn’t use spawn_blocking
persist_tenant_config_*
- uses spawn_blocking with block_on; can switch to tokio::fs instead
heatmap downloader
- uses crashsafe_overwrite , all it does is spawn_blocking

While At It, survey of other spawn_blocking / tokio::fs users

Conclusion

code path	assesment	should bifurcate?
flush_frozen_layer	since `write_to_disk` is already async fn and it’s wrapped in spawn_blocking, this isn’t a problem	NO ACTION
compact_level0_phase1	it’s already on-executor, so, switching to tokio-epoll-uring won’t make it worse	NO ACTION
persist_tenant_config_*	We do this primarily during startup for each tenant ⇒ $POOL_SIZED tokio-epoll-uring system initializations on that code path, but, thread pool doesn’t downscale immediately, so, systems get reused until the work is done.	Not necessary, but would likely benefit from bifurcation.
Also, for the std-fs case, we can avoid the block_on call by switching to a sync version crashsafe_overwrite, will PR that.
heatmap_downloader		Not necessary, but, would benefit. Can switch to sync version of crashsafe_overwrite, will PR that.