How to Upload Binary Data to Server From Javascript
Handling Binary Data — Building a HTTP Server from scratch
On the final mail of BTS: HTTP Server series.
I wrote a barebone HTTP server that can handle requests and respond appropriately.
I call back I covered the basics, only that server is express in what it tin can practise.
It can only handle text-based Requests and Responses... That ways no image or other media exchange.
And then, if the Request or the Response is larger than a KB, I'm out of luck. Again, not not bad for media...
This article is a transcript of a Youtube video I made.
Oh, hey in that location...
That's my challenge for today, refactor my server to handle arbitrarily sized Requests and avoid treating everything equally
text...
If I desire to be able to handle large requests, the first thing I can do is to read the stream in chunks, 1KB at a time
until there'south nothing left to read.
Once I take all of my chunks, I can concatenate them together into one Typed Array. And voila, arbitrarly sized Asking!
const concat = (... chunks ) => { const zs = new Uint8Array ( chunks . reduce (( z , ys ) => z + ys . byteLength , 0 )); chunks . reduce (( i , xs ) => zs . set ( xs , i ) || i + xs . byteLength , 0 ); render zs ; }; const chunks = []; let n ; exercise { const xs = new Uint8Array ( 1024 ); n = expect r . read ( xs ); chunks . push ( xs . subarray ( 0 , n )); } while ( due north === 1024 ); const asking = concat (... chunks );
The 2d challenge is to figure out how much of the data stream is the Request line and the Headers versus the body...
I want to avert reading too far into the body, since information technology might be binary information.
I know that the body starts after the first empty line of the Asking.
Then I could technically, search for the kickoff empty line and and so I'll know that the rest is the body and only parse the first part.
So I wrote this function that will try to find a sequence within the assortment. Get-go tries to find the first occurence of
a byte, and then I tin just test the following bytes until I have a match.
In our case, I desire to find a two CRLF sequences. So I attempt to find the first CR, then check if it is followed by LF, CR
and LF... And, I repeat this until I notice the empty line.
export const findIndexOfSequence = ( xs , ys ) => { let i = xs . indexOf ( ys [ 0 ]); let z = simulated ; while ( i >= 0 && i < xs . byteLength ) { let j = 0 ; while ( j < ys . byteLength ) { if ( xs [ j + i ] !== ys [ j ]) break ; j ++ ; } if ( j === ys . byteLength ) { z = true ; break ; } i ++ ; } return z ? i : null ; };
🐙 You will observe the code for this mail here: https://github.com/i-y-land/HTTP/tree/episode/03
The problem with this approach is that I have to traverse the whole request, and information technology might finish upwardly that the request doesn't
take a body, and therefore I wasted my fourth dimension.
Instead, I will read the bytes 1 line at a time, finding the nearest CRLF and parse them in club.
On the first line, I will excerpt the method and the path.
Whenever I find an empty line, I will assume the is trunk is next and finish.
For the remaining lines, I will parse them as header.
// https://github.com/i-y-country/HTTP/hulk/episode/03/library/utilities.js#L208 consign const readLine = ( xs ) => xs . subarray ( 0 , xs . indexOf ( LF ) + 1 ); export const decodeRequest = ( xs ) => { const headers = {}; let body , method , path ; const n = xs . byteLength ; let i = 0 ; let seekedPassedHeader = false ; while ( i < n ) { if ( seekedPassedHeader ) { body = xs . subarray ( i , north ); i = due north ; continue ; } const ys = readLine ( xs . subarray ( i , n )); if ( i === 0 ) { if ( ! findIndexOfSequence ( ys , encode ( " HTTP/ " ))) break ; [ method , path ] = decode ( ys ). split up ( " " ); } else if ( ys . byteLength === 2 && ys [ 0 ] === CR && ys [ ane ] === LF && xs [ i ] === CR && xs [ i + 1 ] === LF ) { seekedPassedHeader = truthful ; } else if ( ys . byteLength === 0 ) intermission ; else { const [ fundamental , value ] = decode ( ys . subarray ( 0 , ys . indexOf ( CR ) || ys . indexOf ( LF )), ). carve up ( / (?< =^ [ A-Za-z- ] + )\south *: \s */ ); headers [ key . toLowerCase ()] = value ; } i += ys . byteLength ; } return { trunk , headers , method , path }; };
On the other hand, the part to encode the Response is absurdly simpler, I can pretty much apply the function I already made
and just encode the result. The biggest departure, is that I have to exist aware that the body might not
be text and should be kept as a Typed Assortment. I can encode the header and so concat the result with the torso.
// https://github.com/i-y-land/HTTP/blob/episode/03/library/utilities.js#L248 export const stringifyHeaders = ( headers = {}) => Object . entries ( headers ) . reduce ( ( hs , [ central , value ]) => ` ${ hs } \r\northward ${ normalizeHeaderKey ( key )} : ${ value } ` , "" , ); export const encodeResponse = ( response ) => concat ( encode ( `HTTP/1.1 ${ statusCodes [ response . statusCode ]}${ stringifyHeaders ( response . headers ) } \r\n\r\northward` , ), response . body || new Uint8Array ( 0 ), );
From there, I accept plenty to write a simple server using the serve function I've implemented previously.
I can decode the request... then encode the response.
... serve ( Deno . mind ({ port }), ( xs ) => { const request = decodeRequest ( xs ); if ( request . method === " Go " && asking . path === " / " ) { return encodeResponse ({ statusCode : 204 }) } } ). catch (( e ) => console . mistake ( e ));
I could answer to every requests with a file. That is a good start to a static file server.
... if ( request . method === " Get " && request . path === " / " ) { const file = Deno . readFile ( ` ${ Deno . cwd ()} /image.png` ); // read the file return encodeResponse ({ torso : file , headers : { " content-length " : file . byteLength , " content-blazon " : " image/png " }, statusCode : 200 }); }
I can get-go my server and open a browser to visualize the epitome.
With a bit more effort, I can serve any file withing a given directory.
I would attempt to access the file and cross-reference the MIME blazon from a currated list using the extension.
If the arrangement tin can't notice the file, I will render 404 Not Found.
const sourcePath = ( await Deno . permissions . query ({ proper name : " env " , variable : " SOURCE_PATH " })) . state === " granted " && Deno . env . get ( " SOURCE_PATH " ) || ` ${ Deno . cwd ()} /library/assets_test` ; ... if ( request . method === " Go " ) { try { const file = await Deno . readFile ( sourcePath + asking . path ); // read the file return encodeResponse ({ body : file , headers : { " content-length " : file . byteLength , [ " content-type " ]: mimeTypes [ request . path . lucifer ( / (?< extension> \.[ a-z0-9 ] +$ ) / )?. groups ?. extension . toLowerCase () ]. join ( " , " ) || " plain/text " , }, statusCode : 200 }); } catch ( e ) { if ( e instanceof Deno . errors . NotFound ) { // if the file is non constitute return encodeResponse ({ body : new Uint8Array ( 0 ), headers : { [ " Content-Length " ]: 0 , }, statusCode : 404 , }); } throw eastward ; } }
With a broadly similar arroyo, I can receive any file.
const targetPath = ( expect Deno . permissions . query ({ name : " env " , variable : " TARGET_PATH " })) . state === " granted " && Deno . env . go ( " TARGET_PATH " ) || ` ${ Deno . cwd ()} /` ; ... if ( request . method === " Become " ) { ... } else if ( asking . method === " POST " ) { await Deno . writeFile ( targetPath + asking . path , request . body ); // write the file render encodeResponse ({ statusCode : 204 }); }
At present, yous tin can gauge if you wait at the position of your scrollbar that things can't be that unproblematic...
I run across ii problems with my current approach.
I have to load whole files into memory before I can offload it to the File Organization which that can go a bottle neck at
calibration.
Some other surprising event is with file uploads...
When uploading a file, some clients, for example curl volition brand the asking in two steps... The first request is
testing the terrain stating that information technology wants to upload a file of a certain type and length and requires that the server
replies with 100 go on earlier sending the file.
Because of this behaviour I demand to retain access to the connectedness, the writable resources.
And then I think I will take to refactor the serve function from accepting a office that takes a Typed Assortment as an
statement, to a function that takes the connection.
This could also exist positive modify that would facilitate implementing powerful middleware afterwards on...
export const serve = async ( listener , f ) => { for await ( const connection of listener ) { await f ( connection ); } };
There's two means that my server can handle file uploads.
1 possibility is that the customer tries to to post the file direct,
I accept the option to read the header and refuse the asking if information technology'southward too big. The other possibility is that the
client expects me to answer first.
In both case I will read the commencement clamper and then start creating the file with the data processed. Then I want to
to read 1 clamper at a fourth dimension from the connection and systematically write them to the file. This mode, I never hold
more than than 1KB in memory at a time... I practise this until I tin can't read a whole 1KB, this tells me that the file has been
completely copied over.
consign const copy = async ( r , w ) => { const xs = new Uint8Array ( 1024 ); let n ; let i = 0 ; exercise { n = await r . read ( xs ); await westward . write ( xs . subarray ( 0 , north )); i += due north ; } while ( n === 1024 ); return i ; }; ... let xs = new Uint8Array ( 1024 ); const n = wait Deno . read ( r . rid , xs ); const asking = xs . subarray ( 0 , n ); const { fileName } = request . path . friction match ( /.* ?\/(?< fileName> (?:[^ % ] |% [ 0-9A-Fa-f ]{ii}) + \.[ A-Za-z0-9 ] + ?) $/ , )?. groups || {}; ... const file = await Deno . open up ( ` ${ targetPath } / ${ fileName } ` , { create : truthful , write : truthful , }); if ( request . headers . expect === " 100-go along " ) { // write the `100 Continue` response look Deno . write ( connectedness . rid , encodeResponse ({ statusCode : 100 })); const ys = new Uint8Array ( 1024 ); const n = expect Deno . read ( connection . rid , ys ); // read the follow-upwardly xs = ys . subarray ( 0 , n ); } const i = findIndexOfSequence ( xs , CRLF ); // detect the beginning of the body if ( i > 0 ) { await Deno . write ( file . rid , xs . subarray ( i + four )); // write possible file chunk if ( xs . byteLength === 1024 ) { await copy ( connexion , file ); // copy subsequent chunks } } await connection . write ( encodeResponse ({ statusCode : 204 }), // end the exchange ); ...
From in that location, I can rework the part that responds with a file.
Similarly to the two-step request for receiving a file, a customer may opt to asking the headers for a given file
with the Caput method.
Because I want to support this characteristic, I tin first go data from the requested file, then I can offset writing
the headers and only if the request's method is GET -- not HEAD -- I will copy the file to the connectedness.
... attempt { const { size } = look Deno . stat ( ` ${ sourcePath } / ${ fileName } ` ); await connectedness . write ( encodeResponse ({ headers : { [ " Content-Blazon " ]: mimeTypes [ fileName . match ( / (?< extension> \.[ a-z0-9 ] +$ ) / )?. groups ?. extension . toLowerCase () ]. join ( " , " ) || " plain/text " , [ " Content-Length " ]: size , }, statusCode : 200 , }), ); if ( asking . method === " GET " ) { const file = await Deno . open up ( ` ${ sourcePath } / ${ fileName } ` ); await re-create ( file , connectedness ); } } catch ( due east ) { if ( due east instanceof Deno . errors . NotFound ) { Deno . write ( connection . rid , encodeResponse ({ headers : { [ " Content-Length " ]: 0 , }, statusCode : 404 , }), ); } throw e ; } ...
Wow. At this point I have to exist either very confident with my programming skills or sadistic...
I need to implement a slew of integrations tests before going any further.
I created four static files for this purpose, a brusque text file, less than a KB, a longer text file, an image and
music...
For that purpose, I wrote a college-gild-function that will initialize the server before calling the test function.
// https://github.com/i-y-land/HTTP/hulk/episode/03/library/integration_test.js#L6 const withServer = ( port , f ) => async () => { const p = wait Deno . run ({ // initialize the server cmd : [ " deno " , " run " , " --let-all " , ` ${ Deno . cwd ()} /cli.js` , Cord ( port ), ], env : { LOG_LEVEL : " ERROR " , " NO_COLOR " : " one " }, stdout : " nil " , }); look new Promise (( resolve ) => setTimeout ( resolve , k )); // await to be sure try { await f ( p ); // call the exam part passing the process } finally { Deno . close ( p . rid ); } };
With that, I generate a bunch of tests to download and upload files; this ensures that my code is working as expected.
// https://github.com/i-y-land/HTTP/hulk/episode/03/library/integration_test.js#L58 [...] . forEach ( ({ headers = {}, method = " GET " , path , championship , f }) => { Deno . exam ( `Integration: ${ title } ` , withServer ( 8080 , async () => { const response = wait fetch ( `http://localhost:8080 ${ path } ` , { headers , method , }); await f ( response ); }, ), ); }, );
When I got to that point, I realized that my serve function was starting to be very... long.
I knew I needed to refactor it into 2 functions receiveStaticFile and sendStaticFile.
But, because I need to be able to check the Request line to route to the right part, and I can only read the request
once...
I knew that I was in problem.
I need something that can keep part of the data in retention while retaining access to the raw connection...
... if ( method === " Mail service " ) { render receiveStaticFile (?, { targetPath }); } else if ( method === " Go " || method === " HEAD " ) { render sendStaticFile (?, { sourcePath }); } ...
I could have decoded the request and shove the connection in there and telephone call it a solar day...
But it didn't experience correct aaaand I guess I dearest making my life harder.
const request = decodeRequest ( connection ); request . connection = connectedness ; ... if ( method === " Postal service " ) { return receiveStaticFile ( asking , { targetPath }); } else if ( method === " Become " || method === " HEAD " ) { render sendStaticFile ( asking , { sourcePath }); } ...
The solution I came up with was to write a buffer. It would hold in memory only a KB at a time, shifting the bytes
each time I read a new chunk. The advantage of that is I can motion the cursor back to the beginning of the buffer
and read-back parts that I need.
Best of all, the buffer has the same methods as the connexion; and then the 2 could exist used interchangeably.
I won't become into the details because it's a bit dry, but if you want to checkout the code, it'south currently on Github.
// https://github.com/i-y-land/HTTP/blob/episode/03/library/utilities.js#L11 export const factorizeBuffer = ( r , mk = 1024 , ml = 1024 ) => { ... }
With this new toy I can read a chunk from the connection, route the request, motion the cursor dorsum to the kickoff and
laissez passer the buffer to the handler function like goose egg happened.
The peek role specifically has a like signature to read, the difference is that it will move the cursor
back, read a chunk from the buffer in memory and and then finally move the cursor back again.
serve ( Deno . listen ({ port }), async ( connexion ) => { const r = factorizeBuffer ( connection ); const xs = new Uint8Array ( 1024 ); const reader = r . getReader (); await reader . peek ( xs ); const [ method ] = decode ( readLine ( xs )). split ( " " ); if ( method !== " GET " && method !== " POST " && method !== " HEAD " ) { render connection . write ( encodeResponse ({ statusCode : 400 }), ); } if ( method === " Mail " ) { return receiveStaticFile ( r , { targetPath }); } else { return sendStaticFile ( r , { sourcePath }); } } )
To finish this, like a boss, I finalize the receiveStaticFile (https://github.com/i-y-country/HTTP/hulk/episode/03/library/server.js#L15) and sendStaticFile (https://github.com/i-y-land/HTTP/blob/episode/03/library/server.js#L71) functions, taking care of all
the edge cases.
Finally, I run all the integration tests to ostend that I did a skilful job. And uuugh. Sleeeep.
This ane turned out to be a lot more full of surprise than I was prepared for.
When I realized that some client ship file in 2-steps, it really threw a wrench to my plans...
Just it turned out to an amazing learning opportunity.
I really hope that you lot are learning equally much every bit I am.
On the vivid side, this forced me to put together all the tools that I know I will need for the next post.
Next, I want to look into streaming in more details and build some middlewares, starting with a logger.
From there, I am sure that I can tackle building a nice trivial router which will wrap this upward pretty nicely.
All of the code is bachelor on Github, if you lot have a question exercise no hesitate to ask...
Oh speaking of that, I launched a Discord server, if y'all want to join.
🐙 You will find the code for this episode here: https://github.com/i-y-state/HTTP/tree/episode/03
💬 You can join the I-Y community on Discord: https://discord.gg/eQfhqybmSc
At whatever charge per unit, if this commodity was useful to you lot, hit the similar button, exit a annotate to allow me know or best of all,
follow if you haven't already!
Ok adieu now...
Source: https://dev.to/sebastienfilion/building-a-http-server-from-scratch-implementing-file-download-and-upload-438d
0 Response to "How to Upload Binary Data to Server From Javascript"
Post a Comment