{"id":436,"date":"2024-07-16T22:04:38","date_gmt":"2024-07-16T19:04:38","guid":{"rendered":"https:\/\/caupo.ee\/blog\/?p=436"},"modified":"2024-12-19T09:46:26","modified_gmt":"2024-12-19T07:46:26","slug":"net-8-file-upload-streaming-with-chunks-for-larger-files-than-4gb","status":"publish","type":"post","link":"https:\/\/caupo.ee\/blog\/2024\/07\/16\/net-8-file-upload-streaming-with-chunks-for-larger-files-than-4gb\/","title":{"rendered":".NET 8 File upload streaming with chunks for larger files than 4GB!"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"1024\" src=\"https:\/\/caupo.ee\/blog\/wp-content\/uploads\/2024\/07\/Designer.jpeg\" alt=\"\" class=\"wp-image-441\" srcset=\"https:\/\/caupo.ee\/blog\/wp-content\/uploads\/2024\/07\/Designer.jpeg 1024w, https:\/\/caupo.ee\/blog\/wp-content\/uploads\/2024\/07\/Designer-300x300.jpeg 300w, https:\/\/caupo.ee\/blog\/wp-content\/uploads\/2024\/07\/Designer-150x150.jpeg 150w, https:\/\/caupo.ee\/blog\/wp-content\/uploads\/2024\/07\/Designer-768x768.jpeg 768w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Yet again I did not find a good example on the internet for .NET 8 and pure JS solution which combined the two (streaming + chunking upload). So here is my implementation using .NET 8 and pure JS to upload larger than 4GB files chunk by chunk.<\/p>\n\n\n\n<p><strong>FRONTEND:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>const chunkSize = 2097152; \/\/ 2MB\nconst filename = file.name;\n\nlet totalChunks = (file.size \/ chunkSize);\nif (file.size % chunkSize != 0) {\n    totalChunks++;\n}\n\nlet idx = 0;\n\nfor (let start = 0; start &lt; file.size; start += chunkSize) {\n    const chunk = file.slice(start, start + chunkSize)\n    const formData = new FormData();\n    formData.set(\"FileUpload.FormFile\", chunk);\n    await fetchAPI(`\/file\/{uuid}\/chunk\/${idx}`, {\n        method: \"post\",\n        body: formData\n    });\n    idx ++;\n}\n\nconst fileExtension = filename.split(\".\").pop();\nawait fetchAPI(`\/file\/{uuid}\/complete\/${fileExtension}\/${file.type.replace(\"\/\", \"-\")}`, {\n    method: \"post\"\n});<\/code><\/pre>\n\n\n\n<p><strong>BACKEND<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>    &#91;HttpPost(\"chunk\/{chunkIdx}\")]\n    &#91;Authorize]\n    &#91;DisableFormValueModelBinding]\n    &#91;DisableRequestSizeLimit]\n    public async Task&lt;IActionResult&gt; MultiUpload(&#91;FromRoute] Guid uuid, &#91;FromRoute] int chunkIdx, &#91;FromRoute] int chunkSize, CancellationToken cancellationToken)\n    {\n        var request = HttpContext.Request;\n        var reader = GetReader(request, mediaTypeHeader);\n        var section = await reader.ReadNextSectionAsync(cancellationToken);\n\n        while (section != null)\n        {\n            var hasContentDispositionHeader = HasContentDispositionHeader(section, out ContentDispositionHeaderValue? contentDisposition);\n\n            if (IsSectionsDoneAndValid(contentDisposition, hasContentDispositionHeader))\n            {\n                var fileName = $\"part_{chunkIdx}\";\n                var fileSizeLimit = GetFileSizeLimit(configuration);\n                await SaveFile(fileName, fileSizeLimit, section, contentDisposition!, uuid.ToString());\n\n                return Ok(new\n                {\n                    UploadSuccess = true\n                });\n            }\n            section = await reader.ReadNextSectionAsync(cancellationToken);\n        }\n\n        \/\/ If the code runs to this location, it means that no files have been saved\n        throw new ValidationException(\"File data not present in request.\");\n    }\n\n    &#91;HttpPost(\"complete\/{fileExt}\/{fileType}\")] \/\/ file ext and type can be moved in body params... I'm just gonna leave it like this at this moment. \n    &#91;Authorize]\n    public async Task&lt;IActionResult&gt; UploadComplete(&#91;FromRoute] Guid uuid, &#91;FromRoute] string fileExt, &#91;FromRoute] string fileType, CancellationToken cancellationToken)\n    {\n        var targetpath = GetTargetPath(configuration);\n        var path = Path.Combine(targetpath, uuid.ToString());\n        var newpath = Path.Combine(path, uuid.ToString());\n        var newFileNameWithPath = newpath + $\".{fileExt}\";\n\n        if (System.IO.File.Exists(newFileNameWithPath)) \/\/ Lets remove previous file if exists so the old file wouldnt get appended which would break the file.\n        {\n            System.IO.File.Delete(newFileNameWithPath);\n        }\n\n        var filePaths = Directory.GetFiles(path)\n                            .Where(path =&gt; path.Contains(\"part\")) \/\/ Only include files which contain \"part\" name in it which indicates the chunk files.\n                            .OrderBy(path =&gt; path.Length).ThenBy(path =&gt; path); \/\/ Lets ensure the files are in correct order.\n\n        foreach (var item in filePaths)\n        {\n            MergeFiles(newpath, item);\n        }\n\n        var newFileName = uuid.ToString() + $\".{fileExt}\";\n        System.IO.File.Move(newpath, newFileNameWithPath); \/\/ Lets add file extension to merged file.\n\n        \/\/ TODO save data to database or check for viruses with amsi dll here for example...\n        \n        return Ok(new\n        {\n            UploadSuccess = true\n        });\n    }\n\n    private static void MergeFiles(string file1, string file2)\n    {\n        FileStream? fs1 = null;\n        FileStream? fs2 = null;\n        try\n        {\n            fs1 = System.IO.File.Open(file1, FileMode.Append);\n            fs2 = System.IO.File.Open(file2, FileMode.Open);\n            byte&#91;] fs2Content = new byte&#91;fs2.Length];\n            fs2.Read(fs2Content, 0, (int)fs2.Length);\n            fs1.Write(fs2Content, 0, (int)fs2.Length);\n        }\n        catch (Exception ex)\n        {\n            Console.WriteLine(ex.Message + \" : \" + ex.StackTrace);\n        }\n        finally\n        {\n            fs1?.Close();\n            fs2?.Close();\n            System.IO.File.Delete(file2);\n        }\n    }\n\n    private static MultipartReader GetReader(HttpRequest request, MediaTypeHeaderValue? mediaTypeHeader)\n    {\n        var boundary = HeaderUtilities.RemoveQuotes(mediaTypeHeader?.Boundary.Value).Value ?? throw new ValidationException(\"Boundary is null.\");\n        var reader = new MultipartReader(boundary, request.Body);\n        return reader;\n    }\n\n    private static bool HasContentDispositionHeader(MultipartSection section, out ContentDispositionHeaderValue? contentDisposition)\n    {\n        return ContentDispositionHeaderValue.TryParse(section.ContentDisposition,\n                        out contentDisposition);\n    }\n\n    private static bool IsSectionsDoneAndValid(ContentDispositionHeaderValue? contentDisposition, bool hasContentDispositionHeader)\n    {\n        return hasContentDispositionHeader &amp;&amp; contentDisposition is not null &amp;&amp; contentDisposition.DispositionType.Equals(\"form-data\") &amp;&amp;\n                        !string.IsNullOrEmpty(contentDisposition.FileName.Value);\n    }\n\n    private async Task&lt;FileSaveResult&gt; SaveFile(string fileName, long fileSizeLimit, MultipartSection section, ContentDispositionHeaderValue contentDisposition, string subDir = \"\")\n    {\n        var targetFilePath = GetTargetPathFromConfig(configuration);\n        var newDir = Path.Combine(targetFilePath, subDir);\n\n        if (subDir != string.Empty &amp;&amp; !Directory.Exists(newDir))\n        {\n            Directory.CreateDirectory(newDir);\n        }\n\n        var fileExtension = Path.GetExtension(contentDisposition.FileName).ToString();\n        var saveToPath = Path.Combine(targetFilePath, subDir, fileName + fileExtension);\n        long fileSize = 0;\n\n        using (var targetStream = System.IO.File.Create(saveToPath))\n        {\n            fileSize = targetStream.Length;\n\n            if (fileSizeLimit &lt; fileSize)\n            {\n                throw new ValidationException(\"File size too large!\");\n            }\n\n            await section.Body.CopyToAsync(targetStream);\n        }\n\n        return new FileSaveResult  {\n            SavePath = saveToPath,\n            FileSize = fileSize\n        };\n    }<\/code><\/pre>\n\n\n\n<p>Hope this helps someone. If you have any questions then feel free to comment. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>Yet again I did not find a good example on the internet for .NET 8 and pure JS solution which combined the two (streaming + chunking upload). So here is my implementation using .NET 8 and pure JS to upload larger than 4GB files chunk by chunk. FRONTEND: BACKEND Hope this helps someone. If you &#8230; <a title=\".NET 8 File upload streaming with chunks for larger files than 4GB!\" class=\"read-more\" href=\"https:\/\/caupo.ee\/blog\/2024\/07\/16\/net-8-file-upload-streaming-with-chunks-for-larger-files-than-4gb\/\" aria-label=\"More on .NET 8 File upload streaming with chunks for larger files than 4GB!\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/posts\/436"}],"collection":[{"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/comments?post=436"}],"version-history":[{"count":4,"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/posts\/436\/revisions"}],"predecessor-version":[{"id":442,"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/posts\/436\/revisions\/442"}],"wp:attachment":[{"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/media?parent=436"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/categories?post=436"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/caupo.ee\/blog\/wp-json\/wp\/v2\/tags?post=436"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}