The web application has a configurable maximum allowable upload size (see http://forums.iis.net/t/1169846.aspx). This can be configured in the REST web.config file by changing the maxRequestLength and maxAllowedContentLength values (both must be changed).
By default, we configure this to "20480" to allow approximately 20MB uploads. When the limit is exceeded, IIS just drops the request and a 404 is returned.
We do recommend using the file size validator on the form item. The default is 10mb.