Socket programming for file transfer application
8 November 2023
The journey of 40x transfer speed improvement
Jump to solution to see the improvement
About 3 months ago, I am looking to build an app to explore different type of application that can be built with React Native.
After a quick round of idea selection, I decided to build a file transfer app. Part of the consideration being it scratches my own itch as I regularly transfer files between my Android and iOS devices.
1. Introduction
App should works
- Offline
- Send original file without compression / loss of quality
2. Development Journey
Initial attempt was to use bluetooth to achieve that. Since wearables and devices connect and sync data from each other as I experienced it. Plus back in the days before cloud solutions, bluetooth file sharing was working okay.
Bluetooth technology quickly confuses me it's variation like bluetooth vs. bluetooth low energy (BLE), scanning and connecting devices are a big hurdle before getting into transmitting data between devices.
Then, I am looking for feasible alternative and arrived at a WiFi network approach with TCP socket. I am familiar with the paradigm with my experiences working with chat and video streaming applications.
A quick test of the setup with TCP server and client proofs that connecting devices are easier, and sending text data was achieved shortly after.
The first challenge surfaced and it was me asking myself "How to send image using text string?".
Shortly afterward, I integrated an image picker library into the app and utilized the base64 encoding to convert the image into a text string, which will be used to send over the connection.
While sending the image with base64 from one side to the other has no issue, another challenge quickly arise.
"How to know if the file content has finished sending?" This is so that I can call the save function on the receiver, to save the file content.
Browsing through the internet, I understand that socket will continuously listening to data. It is up to me to either close the socket, and the last bit of data will be flushed out of the buffer and I can save the file when the event onClose is triggered.
That works if I am sending a single file, but what if I am sending multiple files (let's say 100) in one go?
Now the alternative is to send a special string as a delimiter to indicate the termination of the file content.
With base64 encoding, @ is surely not part of the character that will exists, therefore it is a good candidate for this use case.
To think about data in socket connection is to imagine it as a continous string of data that keeps notify you when something new arrived.
Now that I know what to look for to split the data with a delimiter (let's call it @END@), here comes the other challenge: knowing where does the file content starts.
There is a need to transmit information other than the file content. For example, information that informs sender to update the UI when the client successfully saved the file, or data that used for chat messages.
With the same approach used to determine the @END@ of a content, now I need to have it for the beginning of a content, (let's call it @HEAD@).
Now we are able to separate a whole chunk of content from start to end by identifying @HEAD@ and @END@, regardless whether the content is a file content, or a stringified string.
When we are sending something in socket, it will not always send it out immediately. There is a concept known as buffer that keeps collecting the data and send it out in one go, and we have no control over when it will send out or wait.
We can imagine it as a waiting area for data to pile up and send out when it is ready.
This behaviour is important to understand because we need to design our system to be able to process the transmitted data that are not guaranteed to be in it's own chunk from the sender's intention.
For instance, before sending any content, I will send the header information to the client. From the client's perspective, the first batch of data received, may comprise of header only data or header and content data.
So the client will need to handle that accordingly.
When everything is finally in place, I am able to send content between iOS and android devices. With different type of content such as stringified JSON for UI updates or text messages and Base64 string for file content that could be any file.
A huge milestone achieved! 🎉
Finally a working file transfer app, from initial ideation which I have no clue how exactly it can be done.
Then comes the next challenge. Transfer speed is slow. Significantly slow with large file, and this is not even multiple files.
At this point, initially I am about to move on to other features that are more interesting to me, like scan for network for available devices to connect. This will greatly reduce the effort to initiate a transfer of file and have better user experience.
However, I chose to work on improving the transfer speed instead. I simply couldn't bear with the slow transfer speed. Despite this means that I probably need to work on native code, which has high degree of uncertainties but I am determined it can be done.
First step of transfer speed optimization begins with revisiting the existing algorithm that process the incoming data from socket connection.
With careful refactoring and simplifying logic in loops, I managed to improve the transfer speed from ~100Kbps to ~300Kbps.
That's 3 times of improvement and have obvious reduction in transfer time. However, the goal is to reach above 1000Kbps because that's what the active incumbents on the market are offering, and I am convinced it can be done, with React Native.
Also because that's what the whole point of starting this project, to explore and push the limit of app built with React Native.
The next junction of optimization leads to which path to take to achieve the goal of 1000Kbps transfer speed.
I could either:
- Optimize Javascript code in hope for the next 3x improvement
- Opt for React Native's new architecture
- Work with native platform code
What I have done was all 3 of them, in the order of the list above.
Well, fast forward today and looking back to the decision, I would still make the same decision if I were to choose again.
There isn't anymore room for optimization working on Javascript code and logic. The bottleneck is the bridge that is used for communication between native and JS code.
I came to this realization after spending more time looking for a simple tweak but unfortunately, no easy fix using this approach.
The next step was to try out the new architecture and hopefully transfer speed will just be faster.
Lots of experimentation: enabling new architecture, learn Codegen, Turbo modules, working with custom JSI binding to send data between native and JS without the bridge, configuring CMakeLists.txt for C++ build, benchmark the performance and so on.
After spending 2 weeks learning the rope, the result is not up to expectation. Enabling the new architecture is not a silver bullet for the problem. No significant improve in speed.
Despite heading into the wrong direction and invested time without fruitful outcome, finally it is now clear to me, with more learning from the internet about the problem, that I can do it with native module, without new architecture.
The next steps is to read and browse through the source code of existing native module and figure out a solution.
The transfer speed improve tremendously, exceeded my initial goal way beyond my expectation. It's mind blowing that it reaches 12,000Kbps in good network condition. 🤯 40x improvement from 300Kbps to 12,000Kbps.
Transferring videos and huge files are way much faster now, and multiple files transfer has no issue as well with fast transfer speed.
I am glad it worked out that way but I am far from done with the application. There are just so many exciting features I wanted to implement. It's time to take a break from this development and it's a joy to use to use this app on a weekly basis to transfer files.
Check out the app and let me know what you think!
Thanks for reading, cheers!