You can only increase the size up to 2GB (maximum limit).
https://docs.tibco.com/pub/spotfire_server/12.4.0/doc/html/TIB_sfire_server_tsas_admin_help/server/topics/spotfire.dxp.worker.web.config_file.html
Extracted from the above document.
=================================
Setting Default Value Description
--------------------------------------------------------------------------------------------------------------
maxReceivedMessageSizeMb 1024 Specify the maximum size, in megabytes (MB),
of files uploaded to the service.
The allowed range is 1 to 2048.
=================================
Unfortunately, in Spotfire, there is currently no way to remove this limitation.