I think it is super cool to think about composable servers, using shared memory blocks, rearrangeable accelerators, and possibly numa cpus. However, I’m not super optimistic who will construct farms of servers in this fashion and use them effectively. Even if you can rearrange your memory and compute capacity, can software really make use of it on the fly? If you have to restart an application to make use of it, then isn’t is simpler to move the application to the hardware, rather than the hardware to the app? Isn’t that the kubernetes, cloud, buzzword future we’ve all been promised? Sure, maybe a few of the supercomputer applications will be able to reserve the hardware in time to use it (batch processing rides again), but I don’t see who else.

The PCI sig needs constant improvements, because they limit the top end of what pcie can do. That said, it’s not clear everyone need adopt newer versions so rapidly.
The consumer space, even high end gamers, probably don’t need a new pcie spec for desktop peripherals nearly so often. Given the cost and trace length limits, it’s not that useful. Usb has largely displaced pcie for most low-perf uses, but there’s no reason a desktop shouldn’t be configured with a single high speed interface, and a few low speed slots.
In the enterprise space, I’m not sure quite what cadence is required for NICs and slot-resident accelerators. Higher than consumer to be sure.
The one I’m not sure about is between chiplets on a package, or at least between sockets on a tightly integrated module. Is Pcie the industry standard QPI now? If yes, then it needs to support those levels of speed.

I’ll say it this way. Because PCI-Express can’t keep up and is not a reasonably good switch that keeps pace with the bus, NVLink Switch had to be invented. And because of that, all accelerators did not have a native, cheap switch architecture from the get-go. I am seeing what would have been possible if this had been done right from the beginning. Also, I think if people could have more bandwidth and more PCI-Express lanes on CPUs today, they would gladly use it. but since this is not happening, PCI switches would be very useful to allow the pooling and sharing of these components. Static configurations are not ideal for either memory or accelerators.

I am thinking forward, when we have CXL memory and accelerators as a standard part of a server. The reason no one uses the switches is because there is an impedance mismatch. The latency of PCI is a hell of a lot less than Ethernet, and I think is a better way to make pods of stuff that is composable.

I could care less about what desktops need. That can’t be the gating factor, and has not been in servers for quite a number of years. If it was, there would not be the kind of GPU compute we currently have in the datacenter, just to give one example. The desktop-first thinking is what has gotten us in this mess as far as I am concerned.

Unless the desktop market needs this, the server/AI market is just going to have to wait, since they don’t drive the numbers. Also, how many servers actually need this as well outside of hyperscalers or research labs? I just don’t see this happening any time soon.

We need a bifurcation in the PCIe version shipped with server and with desktop/notebook CPUs. PCIe5 has driven prices up and expansion options down on the desktop. We used to get 4 or even 5 x8/x16 electrical PCIe3 slots on a mid price mobo, then a pair of x8 PCIe4, and now anything other than the single x16 PCIe5 option is horrifically expensive. Imagine what a mobo will cost once PCIe7 is the standard connect to an AMD or Intel CPU!

Certainly ever higher bandwidth and ever lower latency PCIe is a boon for building data centers and clouds, but the other end of the market is not served well by this at all.

Thanks for the laughs! Bravo!