I'd check the brake booster. It operates off engine vacuum. There is a one-way valve in the fitting screwed into the intake plenum then a hose that runs across the rear of the engine bay to the brake booster. Maybe the hose is leaking.
Link to Centerline's page showing booster, one-way valve and hose.
Link to Centerline's page showing booster, one-way valve and hose.