← All posts
AIDC

Reading AI Training in Switch Counters

Five traffic signatures, what each one looks like in telemetry, and what to do when you spot them.

The ticket said: “Training slow, network appears congested.” Two hours of investigation later, the network team determined there was no congestion. The ML team had switched from tensor parallelism to expert parallelism between two test runs and not told anyone. The traffic pattern had completely changed. Same hardware. Same workload name on the dashboard. Completely different fabric load.

Nothing was actually wrong. Nobody was actually communicating.

Sign up free to keep reading

This post is for members. Create a free account to read the full article, it only takes a few seconds.

Sign up free