miyako

Miyako

A tool to speed up downloading of large amounts of small files

This project originated from a game I played before. It would download a large number of small files when updating the version. However, there was no optimization, so the speed was very slow. Therefore, I thought about whether there is a way to optimize the download scenario of a large number of small files. After a simple idea, I came up with this idea. , since the download speed is slow because there are too many small files, then just put them all into one large file. After all, the download speed of large files can theoretically reach the full bandwidth!

The principle of the project is to directly link a large number of small files into a block file, and then generate a catalog. The client downloads the block file through the catalog and restores the file locally based on the file size and path. Based on the characteristics of HTTP Range , it can also achieve interruption. Click to reconnect and download a file function

How to use

1. Go to Releases to download the latest client and server binary builds

2. Deploy the server

1. Place the binary file in any directory and create an assets folder in the same directory

2. Place the small files that need to be downloaded faster into any folder within the assets folder, such as assets/example

3. Start the server. When the server starts, it will automatically create packaging files for all subfolders in the assets directory and create catalog.json for them.

4. The server prompts Server started at http://0.0.0.0:8080 which means the startup is complete.

3. Download files via client

1. Place binary files into any directory

2. Enter client <CatalogAddr> to start downloading the file. <CatalogAddr> is the download address of catalog.json , which will be automatically generated when the server starts. Assume that your server file path is assets/example , then the catalog address is http://localhost:8080/example/catalog.json

3. Downloaded files will be automatically saved in the download folder

build

Build requires Golang >= 1.22.2

1. Install dependencies

 go mod tidy

2. Build the server

 go build ./cmd/server/

3. Build client

 go build ./cmd/client/

Quote

Import package

go get -u github.com/nijinekoyo/miyako

Package instructions

1. pkg/catalog
   Used to create and parse catalogs, please see the API documentation

2. pkg/block
   Used to generate file blocks and download file blocks, please see the API documentation

license

This project is distributed under the MIT License agreement