Easier workaround (perhaps easier said than done :) ):
HyTC
- Admin Sets A Limit On Number Of Messages (Total) : X
- As Soon As X Number Of Messages Are Recd, The User Profile Has The "Allow Private Messages: updated to No" With Message Displayed Right On category_list.html and also on Messages Section in "Bold Red" that No More Messages Would Be Recd Unless The Messages Are Deleted as soon as user logs in.
- As Soon Message Count goes down below X on account of Deletion of exisitng messages, Allow Private Messages: updated to Yes in the Profile Section.
- As a consequence of 2, you need not have to handled bouncing of PM .... sender would get an "Indication" that Message can't be sent as soon as he/she clicks on Send ...

HyTC