lbm-gpu: stream in place with the AA pattern

The collide and stream passes each moved the whole population field
through two buffers (304 bytes per cell update). The AA pattern (Bailey
et al. 2009; Wittmann et al. 2013) keeps one buffer: even steps collide
each cell in place with its directions swapped, odd steps read a cell's
incoming populations at its neighbours and write the outgoing ones where
the next step reads them, so each thread writes the addresses it read
and streaming happens inside the collision (152 bytes). A cell's
coordinates are computed once and neighbours wrap with a compare and an
add. The per-link wall rules gather their sources before a scatter
writes them, so no link overwrites another's post-collision value, and
the momentum exchange is taken in the gather of the step whose forces
are read back.

Options::streaming picks the layout; Auto, what a solve uses, streams
in place, which measured faster on every device tried: 1.5x on an RTX
4090, 1.85x on the GB10 and 1.45x on an M4 Max at periodic 128^3.
Options::profile times each kernel with timestamp queries for the
throughput test, which now runs both layouts. The verification tests
run in both layouts, with two new ones comparing fields, mass, forces
and heat with the CPU after odd and even step counts across submission
boundaries.

Part of #562.

crates/lbm-gpu/tests/throughput.rs

@@ -1,6 +1,8 @@
 //! Throughput of the GPU backend (ignored; run with
 //! `cargo test --release -p fieldcraft-lbm-gpu --test throughput -- --ignored --nocapture`),
-//! on the cases of `fieldcraft-lbm`'s throughput test.
+//! on the cases of `fieldcraft-lbm`'s throughput test, streaming in place and with two buffers
+//! (what `Streaming::Auto` picks on this device is marked). `FIELDCRAFT_LBM_GPU_PROFILE=1`
+//! adds each kernel's GPU time (timestamp queries; the throughput is then not representative).
 
 #[path = "../../lbm/tests/common/mod.rs"]
 mod common;
@@ -8,7 +10,7 @@ mod common;
 use common::*;
 use fieldcraft_gpu::select::Policy;
 use fieldcraft_lbm::backend::Backend;
-use fieldcraft_lbm_gpu::GpuFlow;
+use fieldcraft_lbm_gpu::{GpuFlow, Options, Streaming};
 use fieldcraft_model::{BoxFace, FaceCondition};
 
 #[test]
@@ -41,16 +43,33 @@ fn cell_updates_per_second() {
             },
         );
         let (_, flow) = prepare(&m, &obstacles);
-        let Ok(mut gpu) = GpuFlow::new(&flow, Policy::from_env().expect("FIELDCRAFT_GPU names a policy")) else {
-            println!("skipping: no GPU");
-            return;
+        let policy = Policy::from_env().expect("FIELDCRAFT_GPU names a policy");
+        let profile = std::env::var("FIELDCRAFT_LBM_GPU_PROFILE").is_ok_and(|v| v == "1");
+        let Ok(auto) = GpuFlow::new(&flow, policy) else {
+            println!("skipping {name}, {l}³: no GPU or not enough memory");
+            continue;
         };
-        // One warm-up chunk, then the timed run: the chunk's steps are one submission.
-        gpu.advance(1).unwrap();
-        let start = std::time::Instant::now();
-        gpu.advance(steps).unwrap();
-        let _ = gpu.mass().unwrap();
-        let s = start.elapsed().as_secs_f64();
-        println!("{name}, {l}³ on {}: {:.1} M cell updates/s", gpu.adapter_name(), (gpu.fluid_cells() as f64 * steps as f64) / s / 1e6);
+        let picked = auto.streams_in_place();
+        drop(auto);
+        for (streaming, in_place) in [(Streaming::InPlace, true), (Streaming::TwoBuffers, false)] {
+            let mut gpu = GpuFlow::with_options(&flow, policy, Options { streaming, profile }).unwrap();
+            // One warm-up chunk, then the timed run: the chunk's steps are one submission.
+            gpu.advance(1).unwrap();
+            let start = std::time::Instant::now();
+            gpu.advance(steps).unwrap();
+            let _ = gpu.mass().unwrap();
+            let s = start.elapsed().as_secs_f64();
+            println!(
+                "{name}, {l}³ on {}, {}{}: {:.1} M cell updates/s (pipelines {:.2} s)",
+                gpu.adapter_name(),
+                if in_place { "in place" } else { "two buffers" },
+                if in_place == picked { " (auto)" } else { "" },
+                (gpu.fluid_cells() as f64 * steps as f64) / s / 1e6,
+                gpu.pipeline_seconds()
+            );
+            for (kernel, t) in gpu.kernel_times().unwrap_or_default() {
+                println!("    {kernel}: {} dispatches, {:.3} ms each", t.dispatches, 1e3 * t.seconds / t.dispatches.max(1) as f64);
+            }
+        }
     }
 }