Sample code using IBM Aspera APIs for various IBM Aspera products and SDKs:
Various programming languages are proposed.
IBM Aspera API documentation (select 24 items per page on bottom).
Aspera Transfer SDK documentation contains code samples.
Video about Transfer SDK
The IBM Aspera Connect SDK github site contains examples about using the Aspera Connect SDK.
IBM Aspera provides two types of APIs:
Client APIs:
SDKs include libraries to be used in applications to transfer files
Server APIs:
REST APIs (with OpenAPI spec) interact with Aspera applications (Faspex, AoC, Node API, COS, etc...)
Depending on the use case, one might use one or (often) several of those APIs.
This repository is structured like this:
web : an example that shows the use of web SDKd: both the Aspera Connect SDK and Aspera HTTP Gateway SDK
app : examples in various languages using the Aspera Transfer SDK and Aspera Applications REST APIs
Inside each language folder, you will find:
README.md : a specific README for the languageMakefile : a makefile to run the samplessrc : source codesrc/utils : helper classessrc/examples : sample programsSample programs will use server addresses and credentials from a YAML configuration file. Once the configuration file is created, sample programs can be run directly.
Unix-like systems: Linux, macOS... a Makefile is provided to run the samples.
Windows: Refer to Quick start (Windows) below. make might not be available. Use the Makefile as a reference to execute the commands manually.
See Running sample programs.
On the first execution of make: the Transfer SDK will be automatically downloaded.
To download the SDK, execute: make sdk.
Refer to Configuration File: Copy the file config/config.tmpl into private/config.yaml and fill values.
md private
copy configconfig.tmpl privateconfig.yaml
Set the parameter misc.platform to windows-x86_64
Edit required parameters in private/config.yaml, for example Faspex connection information.
Note: Yes, you can also drag and drop, and click, and copy/paste, and edit the file with Notepad, etc...
Prepare the SDK folder
md tmp
Download the Aspera Transfer SDK (here) and extract its contents to the folder identified by sdk_dir in config/paths.yaml :
Note: Make sure that files identified in
config/paths.yamlare in the extracted folder as expected. For example, the following file must exist:/tmp/transfer_sdk/bin/asperatransferd
Run the samples: see Running sample programs
Create a configuration file as specified in Configuration file. Not all values are required, only those needed for the examples you want to run.
For example to execute an individual sample, use make .tested/:
$ cd app/python
$ make list
server aoc faspex faspex5 node shares node_v2
$ make .tested/faspex5Running examples requires the Transfer SDK daemon asperatransferd to be downloaded, and some tools to compile the proto file of the transfer SDK, see Transfer SDK.
For details, refer to the recipe in the Makefile of each language.
A template configuration file is provided: config/config.tmpl.
Copy the file config/config.tmpl into private/config.yaml and fill with your own server addresses, credentials and parameters.
cp config/config.tmpl private/config.yaml
vi private/config.yamlNote: Although the format may look like the configuration file for
ascli, a configuration file forascliis not compatible with this one. You must create a new one.
Set the parameter misc.platform to the architecture used:
osx-arm64osx-x86_64windows-x86_64linux-x86_64linux-s390linux-arm64linux-ppc64leaix-ppc64The parameter trsdk.url can be set to grpc://127.0.0.1:55002 (specify the local port that sdk will use).
Section httpgw is used by the web example only.
Other sections are used by the various examples.
For example, if you want to test only the COS transfer using the Transfer SDK, you can fill the cos section only and leave other sections empty.
Example (with random credentials):
misc:
platform: osx-x86_64
level: debug
transfer_regular: true
trsdk:
url: grpc://127.0.0.1:55002
level: trace
ascp_level: trace
web:
port: 9080
httpgw:
url: https://1.2.3.4/aspera/http-gwy
server:
user: aspera
pass: demoaspera
url: ssh://demo.asperasoft.com:33001
file_download: /aspera-test-dir-small/10MB.1
folder_upload: /Upload
node:
url: https://node.example.com:9092
verify: false
user: node_user
pass: _the_password_here_
folder_upload: /Upload
faspex:
url: https://faspex.example.com/aspera/faspex
user: faspex_user
pass: _the_password_here_
cos:
endpoint: https://s3.eu-de.cloud-object-storage.appdomain.cloud
bucket: my_bucket
key: _the_key_here_
crn: 'crn:v1:bluemix:public:cloud-object-storage:global:_the_crn_::'
auth: https://iam.cloud.ibm.com/identity/token
coscreds:
bucket: mybucket
service_credential_file: ./service_creds.json
region: eu-de
aoc:
org: acme
user_email: [email protected]
private_key: /path/to/my_aoc_key
client_id: aspera.global-cli-client
client_secret: frpmsRsG4mjZ0PlxCgdJlvONqBg4Vlpz_IX7gXmBMAfsgMLy2FO6CXLodKfKAuhqnCqSptLbe_wdmnm9JRuEPO-PpFqpq_Kb
workspace: Default
shared_inbox: TheSharedInboxNote: Sections with HTTPS URLs have a parameter
verify. Set it tofalseto disable server certificate validation for development environments.
Some relative paths are defined in config/paths.yaml (keep those values intact).
The following log levels can be set:
misc.level: sample code log level: debug info warning error
trsdk.level: asperatransferd log level: trace debug info warning error fatal panic
trsdk.ascp_level: ascp log level: trace debug info
Some examples support setting port to 0 (zero) in trsdk.url to use a random port.
Sample application generate a file asperatransferd.conf provided to the transfer sdk daemon, log level there are taken from the general yaml config file.
The Transfer SDK is a gRPC service that allows you to transfer files in an application. It is a client API that can be used in various languages.
The file transfer.proto describes in the remote procedure call interface provided by the daemon asperatransferd.
+----------------+
+ transfer.proto +
+----------------+
|
[protoc]
|
v
+----------------------+ +------------+
+ generated stub code + + your code +
+----------------------+ +------------+
| [combine] |
+-----+----------------------+
|
v
+------------+ +---------------------+
| client app |-----[connect to]---->| Transfer SDK daemon |
+------------+ +---------------------+
| ^ | [executes]
+-------------[executes]----------------+ v
| +------+
[or other method, systemd, or manual]---[executes]------+ | ascp |
+------+
Client applications must use the client source files generated from the transfer.proto file.
Generated (stub) code is provided for convenience in the Transfer SDK for several languages.
It can be used directly or the developer may choose to generate them from the transfer.proto file.
For production and future compatibility it is recommended to generate the stub code from the transfer.proto file.
If you generate stub code yourself, then you can benefit from support to latest platforms and versions.
Most samples here generate the stub code from the transfer.proto file.
Refer to GRPC web site for instructions on how to generate the code.
Sample programs use helper classes located in package utils:
Configuration reads configuration parameters from config.yaml so that it is easier to run any samples.TransferClient creates a configuration file and starting the Transfer SDK daemon: asperatransferd
Rest for simple API calls on Rest APIs.The Transfer SDK requires the following runtime files:
asperatransferd : executable that provides the gRPC serviceascp : executable that actually transfers the filesascp4 : another version of ascpasync : executable for async operationslibafwsfeed : a library for ascp for web socketsaspera-license : the license file for ascp (free use)Optional files:
aspera.conf : the configuration file for ascp
product-info.mf : XML file with information on SDK versionaspera.confThis file is optional for ascp when used in client mode.
The very minimum content is:
<CONF/>It is possible to set some client parameters, like:
xml version='1.0' encoding='UTF-8'?>
<CONF version="2">
<default>
<file_system>
<storage_rc><adaptive>trueadaptive>storage_rc>
<resume_suffix>.aspera-ckptresume_suffix>
<partial_file_suffix>.partialpartial_file_suffix>
<replace_illegal_chars>_replace_illegal_chars>
file_system>
default>
CONF>asperatransferd is a daemon that must be started before using the Transfer SDK.
It drives the transfer of files between two endpoints using embedded ascp.
The client app will connect to it using gRPC on the port specified.
The way to start the daemon is not specified in the SDK. Developers have the choice to start it manually in a separate terminal, or to create a static configuration file and start it using another method (for example, a systemd service).
Examples provided here start the daemon using the TransferClient class.
When asperatransferd starts, if no configuration file is provided with option --config, then it expects to find ascp, ascp4, async, libafwsfeed, aspera-license in specific folders.
In order to place all file in the same folder, then the configuration file must be provided and folders must be set.
The Makefile provided in the samples downloads the SDK and extracts it in a single folder, then the examples generate the configuration file accordingly.
Refer to the HSTS documentation to create a user and get the credentials.
Typically, a node api user is created like this:
/opt/aspera/bin/asnodeadmin -a -u my_node_username -p my_node_password -x my_transfer_userNote: Access key credentials (id and secret) can also be used for the node api user.
Shares provides the following APIs:
/node_api The same examples as for Node API can be used for Shares.
For Aspera on Cloud, several configuration items are required:
org : The AoC Organization, i.e. the name before .ibmaspera.com in the URLuser_email : The user's IBMidprivate_key : The path to the PEM file containing the user's private key. The user configured the associated public key in his AoC User's profile.client_id : (see below) The client app identifierclient_secret : (see below) The client app secretclient_id and client_secret can be:
aspera CLI :
aspera.global-cli-clientfrpmsRsG4mjZ0PlxCgdJlvONqBg4Vlpz_IX7gXmBMAfsgMLy2FO6CXLodKfKAuhqnCqSptLbe_wdmnm9JRuEPO-PpFqpq_KbFor example to extract the ones of Aspera Connect (Drive): strings asperaconnect|grep -B1 '^aspera.drive$'
To test transfers to COS, you will need:
This is the default in the example.
Or it is also possible to use:
private/service_creds.json, follow: get service credentials
