Sounds exactly like turbo lag to me. Some manufacturers get around it
by adding a small turbo (in addition to the regular big one) that spins
up quickly despite the slower exhaust velocity at lower rpm. Audi on
their 2.7T uses two of the same size, but that engine has enough
displacement to provide decent torque even at low rpm. And with each
turbo filling just one bank of 3 cylinders, each one can be small enough
to compress air at lower rpm while being big enough to compress air at
the redline. Engine design is full of compromises.

When you say you 'start of spirited' I'm guessing you use some clutch
slip to get and then keep the engine near 3000 rpm - until the clutch
fully engages and you floor the throttle. If you don't, that's one way
to increase the exhaust velocity enough so the smallish motor doesn't
spend much time fighting to breathe well. You'll typically get the best
acceleration numbers this way. It's also an excellent way to decrease
the life of your clutch if done poorly.