Multicast SDK

Introduction

The Multicast SDK enables applications to connect and interface with the Marlin Network and serves as a common building block for any developer needing to use the Marlin Network for realtime communication between their nodes.

What does the Multicast SDK give you?

PubSub abstraction

The PubSub (short for Publish/Subscribe) abstraction is an extremely powerful and flexible abstraction for multicast communication modelled around the concept of message channels. Subscribers register interest in certain channels and publishers publish relevant messages in these channels which are automatically pushed to registered subscribers.

Adaptability

The Multicast SDK can discover nodes in the Marlin Network, select good peers and automatically establish PubSub channels with them. The application can thus adapt to changes in the Marlin Network without intervention from the developer or the user.

Responsiveness

The Multicast SDK is fully asynchronous utilizing non-blocking networking APIs. This enables developers to build responsive applications with minimal effort.

SDK Concepts

SDK Concepts

Transports

The transport abstraction models a communication channel between two entities using which one side can send/receive data to/from the other side.

A transport has the following attributes -

Common transport types

Transports can be divided into types based on the communication semantics that they provide.

Datagram based transports

Usually send and receive small packets of data.

Sample interface:

struct DatagramTransport {
    SocketAddress src;
    SocketAddress dst;

    void send(Buffer &&packet);
    void did_recv(Buffer &&packet);
    void close();
};

Example: UDP transport

Stream based transports

Usually send and receive byte streams.

Sample interface:

struct StreamTransport {
    SocketAddress src;
    SocketAddress dst;

    void send(Buffer &&bytes);
    void did_recv(Buffer &&bytes);
    void close();
};

Example: TCP transport

Message based transports

Usually send and receive delimited messages.

Sample interface:

struct MessageTransport {
    SocketAddress src;
    SocketAddress dst;

    void send(Buffer &&message);
    void did_recv(Buffer &&message);
    void close();
};

Example: Length-prefix framed transport

Higher-order transports
Transports of a particular type can also be built by wrapping another type of transport. For example, StreamTransport can provide stream semantics by wrapping a base transport providing datagram semantics. See the Higher-order transports section for more info. by wrapping a base transport providing datagram semantics. 

Reusable transports

We can make our transports reusable by isolating the application-specific parts and using behaviour injection to modify them based on application requirements. Behaviour injection can take any of the following forms:

Inheritance

Isolate the application-specific parts into their own functions. Subclass and override these functions as needed.

struct Transport {
    SocketAddress src;
    SocketAddress dst;

    void send(Buffer &&data);
    void close();

    void did_recv(Buffer &&data); // Can subclass and override
};

Not that flexible. Makes writing higher-order transports difficult.

Callbacks

Isolate the application-specific parts into their own callbacks. Set these callbacks as needed.

struct Transport {
    SocketAddress src;
    SocketAddress dst;

    void send(Buffer &&data);
    void close();

    typedef void (*RecvFunc)(Buffer &&data);
    RecvFunc did_recv; // Can set
};

More of a C paradigm than C++. The callbacks need to be globally and statically addressible which imposes significant restrictions on what can be set as a callback(crucially, no normal member functions, only static ones).

Delegates

Outsource the application-specific parts to an external object. The object can be set as needed.

struct TransportDelegate {
    void did_recv(Buffer &&data); // Can implement custom processing
};

template<typename DelegateType>
struct Transport {
    SocketAddress src;
    SocketAddress dst;

    void send(Buffer &&data);
    void close();

    DelegateType *delegate; // Can set
};

We use the delegate pattern for its flexibility and usability while remaining performant. It makes it easy to define a contract between a Transport and its delegate and as a bonus, makes the design easily portable to languages which have enforced contracts(Go interfaces, Rust traits, etc).

Event notifications

Now that we have a delegate pattern in place, we can use it to notify the delegate of significant events occuring in the transport.

Sample interface:

struct TransportDelegate {
    void did_close(); // Notify transport close
};

template<typename DelegateType>
struct Transport {
    DelegateType *delegate;

    void close() {
        ...
        delegate->did_close();
        ...
    }
};

Better delegates

In the delegates that we have above, there are significant design deficiencies:

We can fix this by simply passing the transport as a parameter to the delegate functions:

struct TransportDelegate {
    void did_recv(Transport<TransportDelegate> &transport, Buffer &&data) {

        // Can implement custom processing based on transport attributes
        if(transport.dst == X) {
            ...
        }

        // Can directly respond
        transport.send(response_data);
    }
};

template<typename DelegateType>
struct Transport {
    DelegateType *delegate;

    void recv_cb() {
        ...
        delegate->did_recv(*this, data);
        ...
    }
};

Canonical transports

Based on the design choices made above, our transports(and delegates) usually look something like this:

struct TransportDelegate {
    // Notify data sent
    void did_send(Transport<TransportDelegate> &transport, Buffer &&data);

    // Notify data receive
    void did_recv(Transport<TransportDelegate> &transport, Buffer &&data);

    // Notify successful dial
    void did_dial(Transport<TransportDelegate> &transport);

    // Notify close
    void did_close(Transport<TransportDelegate> &transport);
};

template<typename DelegateType>
struct Transport {
    // Source
    SocketAddress src;

    // Destination
    SocketAddress dst;

    // Delegate
    DelegateType *delegate;

    // Send data to dst
    void send(Buffer &&data);

    // Close the transport
    void close();
};

Higher-order transports

Higher-order transports are built by wrapping another transport as a base and customizing its behaviour. Given the use of the delegate pattern, we can simply insert the higher-order transport as a delegate for the base transport and intercept all delegate calls to modify the behaviour as needed.

Example - StreamTransport

Our higher-order transports usually look something like this:

template<typename DelegateType, template<typename> class SomeTransport>
struct HigherOrderTransport {
    // Source
    SocketAddress src;

    // Destination
    SocketAddress dst;

    // Delegate
    DelegateType *delegate;

    // Base transport
    typedef SomeTransport<HigherOrderTransport<DelegateType, SomeTransport>> BaseTransport;

    BaseTransport &transport;

    // Send data to dst
    // Will eventually call transport.send to actually send the data
    void send(Buffer &&data);

    // Close the transport
    // Will eventually call transport.close to actually close the transport
    void close();

    //-------- BaseTransport delegate functions below --------//

    // Intercept data sent and do custom processing if needed
    // Will eventually call delegate->did_send to notify
    void did_send(BaseTransport &transport, Buffer &&data);

    // Intercept data receive and do custom processing if needed
    // Will eventually call delegate->did_recv to notify
    void did_recv(BaseTransport &transport, Buffer &&data);

    // Intercept successful dial and do custom processing if needed
    // Will eventually call delegate->did_dial to notify
    void did_dial(BaseTransport &transport);

    // Intercept close and do custom processing if needed
    // Will eventually call delegate->did_close to notify
    void did_close(BaseTransport &transport);
};
SDK Concepts

Transport Factories

Transports are created by transport factories in one of two ways:

Sample interface:

struct TransportFactory {
    // Set source address
    void bind(SocketAddress &addr);

    // Start listening for incoming dials
    void listen();

    // Dial destination address
    void dial(SocketAddress &addr);
};

Higher-order transport factories
Higher-order transport factories complement higher-order transports by wrapping a base transport factory. For example, StreamTransportFactory produces stream transports by wrapping around a base factory which produces datagram transports.

SDK Concepts

Endianness

Endianness refers to the ordering of components in a representation of an entity. For our purposes, we use it to refer to the ordering of bytes in multi-byte integers.

Overview

Let's look at how memory allocation and storage of multi-byte integers work. For illustration, we consider 32-bit integers. When you wish to store a 32-bit integer, the program allocates 4 bytes worth of memory.

Now we have a choice on how we want to store the integer - the 4 bytes in the integer can be mapped to the 4 bytes of memory in any order giving us 24 possible endian options. As long as we use the same ordering while storing and retrieving any integer, the program can function without error.

Big endian

Big endian refers to ordering by most significant byte to least. A 32-bit integer 0x12345678 would be stored as

+++++++++++++++++++++++++++++
| 0x12 | 0x34 | 0x56 | 0x78 |
+++++++++++++++++++++++++++++

Example systems: TCP, IBM z/Architechture

Little endian

Little endian refers to ordering by least significant byte to most. A 32-bit integer 0x12345678 would be stored as

+++++++++++++++++++++++++++++
| 0x78 | 0x56 | 0x34 | 0x12 |
+++++++++++++++++++++++++++++

Example systems: Intel/AMD x86-64, RISC-V

Why should I care?

Endianness is crucial while serializing and deserializing data transmitted over the network between systems with different orderings. Consider the following serialization and deserialization steps happening in a big and little endian system respectively:

// Serialization in big endian system
char buf[4]; uint32_t i = 0xff;
memcpy(buf, &i, 4);

// Transmitted through the network as | 0x00 | 0x00 | 0x00 | 0xff |

// Deserialization in little endian system
char buf[4] = { 0x00, 0x00, 0x00, 0xff }; uint32_t i;
memcpy(&i, buf, 4);

// i now contains | 0x00 | 0x00 | 0x00 | 0xff | in memory
// which is 0xff000000 in the little endian system
// Very different from 0xff!!!

A mismatch in endianness causes the data read to be widly different. Best practice is to fix an endianness for the transmitted data (called network ordering) and handle conversions in the clients based on their endianness (called host ordering).

SDK Concepts

PubSub

PubSub (short for Publish/Subscribe) is an abstraction for a messaging system where senders don't send the message directly to a specific receiver. Instead, messages are grouped into channels that interested receivers (also called subscribers) can subscribe to. Senders (also called publishers) publish messages to there channels which are then independently transmitted to the receivers.

Widely used examples of PubSub systems include WebSub, Kafka and IGMP.

Advantages

Modular

The sender and receiver of messages are decoupled - the publisher doesn't know nor care about which subscribers receive his message and the subscriber doesn't know nor care about which publisher sent the message that he just received. They simply care about channels and are ignorant about the underlying mechanisms for message delivery.

Dynamic

Publishers and subscribers can come and go without affecting the system since they run independently. The messaging system can adapt to changes and keep things running smoothly without interruptions in service.

Scalable

The system has the potential to scale much bigger than traditional client-server messaging systems by taking advantage of better network topology, better routing and message caching among other things.

SDK Tutorial

SDK Tutorial

Overview

In this tutorial, you will learn how to use the Marlin Multicast SDK to send and receive information using the Marlin Network.

Prerequisites

This tutorial assumes familiarity with using the command line. You will need a C++ compiler (supporting C++17), CMake (atleast 3.13 or higher) and Make installed on your computer.

Contents

You will be guided through the following parts:

Code samples

To make the tutorial easier to follow or if you get stuck somewhere, you can find code samples for each part here.

SDK Tutorial

Part 1: Setting up a basic C++ project using CMake and Make for building

Step 1 - Create a working directory

Open a terminal, navigate to a suitable directory and run the following commands:

$ mkdir tutorial
$ cd tutorial

This will create an empty directory called tutorial inside which we'll be working.

Step 2 - Add a minimal C++ program

Create a new file in tutorial called main.cpp and fill it with the following piece of code:

int main() {
    return 0;
}

The above is a small C++ program with a basic main function that doesn't do anything (for now).

Step 3 - Add a CMakeLists.txt file

Create a new file in tutorial called CMakeLists.txt.

Set the minimum CMake version required to 3.13. While we don't need such a high CMake version right now, we'll need it in later parts of the tutorial.

cmake_minimum_required(VERSION 3.13 FATAL_ERROR)

Set project details, primarily the language to let CMake know that this is a C++ project.

project(tutorial VERSION 0.0.1 LANGUAGES CXX)

Add a new executable target with our main program above.

add_executable(tutorial
    main.cpp
)

Once you have followed the above instructions, you should have a CMakeLists.txt that looks like this:

cmake_minimum_required(VERSION 3.13 FATAL_ERROR)
project(tutorial VERSION 0.0.1 LANGUAGES CXX)

add_executable(tutorial
    main.cpp
)

Step 4 - Build

Create a new build directory in tutorial for running any build commands. It is a good practice to follow so that any files produced as a result of building do not pollute the original project.

$ mkdir build
$ cd build

Run CMake to generate a Makefile.

$ cmake .. -DCMAKE_BUILD_TYPE=Release

Run Make to build the executable.

$ make

Check if everything above worked without errors.

Step 5 - Run executable

There should be an executable named tutorial in the build directory. It does nothing for now, but we can still run it using

./tutorial

Conclusion

In this part, we learnt how to create a new C++ project that uses CMake and Make for building. In the next part we will learn how to add the Marlin Multicast SDK to the project using CMake.

SDK Tutorial

Part 2: Adding the Marlin Multicast SDK to the project

Step 1 - Add CMake module to find the SDK and download if not present

Create a folder called cmake in the tutorial directory by running the folling command from inside the build directory.

$ mkdir ../cmake

Create a file called marlin-multicastsdk.cmake inside the cmake folder with the following contents.

find_package(marlinMulticastSDK QUIET)
if(NOT marlinMulticastSDK_FOUND)
    message("-- marlinMulticastSDK not found. Using internal marlinMulticastSDK.")
    include(FetchContent)
    FetchContent_Declare(marlinMulticastSDK
        GIT_REPOSITORY https://gitlab.com/marlinprotocol/marlin.cpp.git
        GIT_TAG master
    )

    # Check if population has already been performed
    FetchContent_GetProperties(marlinMulticastSDK)
    string(TOLOWER "marlinMulticastSDK" lcName)
    if(NOT ${lcName}_POPULATED)
        # Fetch the content using previously declared details
        FetchContent_Populate(marlinMulticastSDK)

        # Bring the populated content into the build
        add_subdirectory(${${lcName}_SOURCE_DIR} ${${lcName}_BINARY_DIR} EXCLUDE_FROM_ALL)
    endif()
else()
    message("-- marlinMulticastSDK found. Using system marlinMulticastSDK.")
endif()

This snippet first checks if the SDK is installed already in the system or is downloaded already. If not, it uses the CMake FetchContent module to download the SDK from version control.

Step 2 - Include the SDK in the build

Add the following line to CMakeLists.txt:

include("${CMAKE_CURRENT_LIST_DIR}/cmake/marlin-multicastsdk.cmake")

This will include the above module in the CMake build process.

To use the SDK in our code, we must first link it to our original executable.

Once you have followed the above instructions, you should have a CMakeLists.txt that looks like this:

cmake_minimum_required(VERSION 3.13 FATAL_ERROR)
project(tutorial VERSION 0.0.1 LANGUAGES CXX)

add_executable(tutorial
    main.cpp
)

# marlinMulticastSDK
include("${CMAKE_CURRENT_LIST_DIR}/cmake/marlin-multicastsdk.cmake")
target_link_libraries(tutorial PUBLIC marlin::multicastsdk)

Step 4 - Build and run executable

Build using

$ cmake .. -DCMAKE_BUILD_TYPE=Release
$ make

and run using

$ ./tutorial

Building should have worked and the executable should run (and still do nothing for now).

Conclusion

In this part, we learnt how to add the Marlin Multicast SDK to the project using CMake. In the next part we will learn how to use the Marlin Multicast SDK to connect to the Marlin Network.

SDK Tutorial

Part 3: Connecting to the Marlin Network

Step 1 - Download and run a Goldfish node

For the purposes of this tutorial, we will use an executable called Goldfish that emulates the Marlin Network locally.

The executable is available here for Linux and here for macOS.

After downloading the executable to the build folder, open a new console and run it using

$ ./goldfish  # You might have given it a different name during download

You might have to change permissions on the file to make it executable using

$ chmod +x ./goldfish  # Add sudo in the beginning if necessary

You might get a verification error on macOS 10.15 that looks something like this: verify_error.pngTo fix this, navigate to System Preferences -> Security & Privacy -> General and click "Allow Anyway" against the goldfish executable:settings_allow.png settings_allow

Check that it prints the IP and ports that it is using in the console.

Step 2 - Set up a multicast client

Include the header for the multicast client and the relevant namespaces using

#include <marlin/multicast/DefaultMulticastClient.hpp>

using namespace marlin::multicast;
using namespace marlin::net;

Before we create the client, we need to create a delegate that is notified of important events by the client (e.g. new messages). See the reference documentation for further information on the events that can be listened to.

class MulticastDelegate {
public:
    void did_recv_message(
        DefaultMulticastClient<MulticastDelegate> &client,
        Buffer &&message,
        Buffer &&witness,
        std::string &channel,
        uint64_t message_id
    ) {}

    void did_subscribe(
        DefaultMulticastClient<MulticastDelegate> &client,
        std::string &channel
    ) {
        SPDLOG_INFO("Did subscribe to channel {}", channel);
    }

    void did_unsubscribe(
        DefaultMulticastClient<MulticastDelegate> &client,
        std::string &channel
    ) {
        SPDLOG_INFO("Did unsubscribe from channel {}", channel);
    }
};

Currently, the delegate ignores all messages and prints subscription/unsubscription logs.

Inside the main function, create a keypair that is used by the client for identification.

    uint8_t static_sk1[crypto_box_SECRETKEYBYTES];
    uint8_t static_pk1[crypto_box_PUBLICKEYBYTES];
    crypto_box_keypair(static_pk1, static_sk1);

Inside the main function, initialize a multicast client and delegate. The client can be customized by providing options during initialization. For further information on the available options, please refer to the reference documentation.

    MulticastDelegate del;
    DefaultMulticastClientOptions clop1 {
        static_sk1,  // Secret key
        {"goldfish"},  // Pubsub channels
        "127.0.0.1:9002"  // Beacon address from the goldfish node
    };

    DefaultMulticastClient<MulticastDelegate> cl1(clop1);
    cl1.delegate = &del;

Step 3 - Run the event loop

The Marlin Multicast SDK is fully asynchronous and uses event loops to acheive this.

Run the event loop using

    return DefaultMulticastClient<MulticastDelegate>::run_event_loop();

Once you have followed the above instructions, you should have a main.cpp that looks like this:

#include <marlin/multicast/DefaultMulticastClient.hpp>


using namespace marlin::multicast;
using namespace marlin::net;

class MulticastDelegate {
public:
    void did_recv_message(
        DefaultMulticastClient<MulticastDelegate> &client,
        Buffer &&message,
        Buffer &&witness,
        std::string &channel,
        uint64_t message_id
    ) {}

    void did_subscribe(
        DefaultMulticastClient<MulticastDelegate> &client,
        std::string &channel
    ) {
        SPDLOG_INFO("Did subscribe to channel {}", channel);
    }

    void did_unsubscribe(
        DefaultMulticastClient<MulticastDelegate> &client,
        std::string &channel
    ) {
        SPDLOG_INFO("Did unsubscribe from channel {}", channel);
    }
};

int main() {
    uint8_t static_sk1[crypto_box_SECRETKEYBYTES];
    uint8_t static_pk1[crypto_box_PUBLICKEYBYTES];
    crypto_box_keypair(static_pk1, static_sk1);

    MulticastDelegate del;

    DefaultMulticastClientOptions clop1 {
        static_sk1,
        {"goldfish"},
        "127.0.0.1:9002"
    };

    DefaultMulticastClient<MulticastDelegate> cl1(clop1);
    cl1.delegate = &del;

    return DefaultMulticastClient<MulticastDelegate>::run_event_loop();
}

This sets up a multicast client that connects to the Marlin Network on 127.0.0.1:9002 can send and receive on the goldfish channel. The client automatically talks to the Goldfish node we ran above and sets up PubSub on the given channel. The client calls the corresponding functions in the delegate to notify it of any significant events.

Step 4 - Build and run executable

Build using

$ cmake .. -DCMAKE_BUILD_TYPE=Release
$ make

and run using

$ ./tutorial

Building should have worked and the executable should run. You should see subscription related logs printed here indicating a successful connection between the client and the Marlin Network.

Conclusion

In this part, we learnt how to use the Marlin Multicast SDK to connect to the Marlin Network. In the next part we will learn how to use the Marlin Multicast SDK to send and receive messages using the Marlin Network.

SDK Tutorial

Part 4: Sending and receiving messages using the Marlin Network

Step 1 - Run two clients

Create second key pair.

    uint8_t static_sk2[crypto_box_SECRETKEYBYTES];
    uint8_t static_pk2[crypto_box_PUBLICKEYBYTES];
    crypto_box_keypair(static_pk2, static_sk2);

Add a second client inside main.cpp.

    DefaultMulticastClientOptions clop2 {
        static_sk2,
        {"goldfish"},
        "127.0.0.1:7002",
        "127.0.0.1:7000"
    };
    DefaultMulticastClient<MulticastDelegate> cl2(clop2);
    cl2.delegate = &del;

This time, we also set options that changes the ports used by the client since the default ports are taken up by the first client.

Step 2 - Handle new messages

Handle new messages inside the did_recv_message function in the delegate. For now, we simply log the messages to the console.

        SPDLOG_INFO(
            "Did recv message: {}",
            std::string(message.data(), message.data() + message.size())
        );

Step 3 - Send messages on some event

In real world scenarios, new messages are usually triggered by external events like timers, other messages, user actions, etc. For the purpose of this tutorial, we take advantage of the client calling did_subscribe to trigger a new message. Add the following to did_subscribe

        client.ps.send_message_on_channel(channel, "Hello!", 6);

Once you have followed the above instructions, you should have a main.cpp that looks like this:

#include <marlin/multicast/DefaultMulticastClient.hpp>


using namespace marlin::multicast;
using namespace marlin::net;

class MulticastDelegate {
public:
    void did_recv_message(
        DefaultMulticastClient<MulticastDelegate> &client,
        Buffer &&message,
        Buffer &&witness,
        std::string &channel,
        uint64_t message_id
    ) {
        SPDLOG_INFO(
            "Did recv message: {}",
            std::string(message.data(), message.data() + message.size())
        );
    }

    void did_subscribe(
        DefaultMulticastClient<MulticastDelegate> &client,
        std::string &channel
    ) {
        SPDLOG_INFO("Did subscribe to channel {}", channel);
        client.ps.send_message_on_channel(channel, "Hello!", 6);
    }

    void did_unsubscribe(
        DefaultMulticastClient<MulticastDelegate> &client,
        std::string &channel
    ) {
        SPDLOG_INFO("Did unsubscribe from channel {}", channel);
    }
};

int main() {
    uint8_t static_sk1[crypto_box_SECRETKEYBYTES];
    uint8_t static_pk1[crypto_box_PUBLICKEYBYTES];
    crypto_box_keypair(static_pk1, static_sk1);

    uint8_t static_sk2[crypto_box_SECRETKEYBYTES];
    uint8_t static_pk2[crypto_box_PUBLICKEYBYTES];
    crypto_box_keypair(static_pk2, static_sk2);

    MulticastDelegate del;

    DefaultMulticastClientOptions clop2 {
        static_sk2,
        {"goldfish"},
        "127.0.0.1:9002",
        "127.0.0.1:7002",
        "127.0.0.1:7000"
    };
    DefaultMulticastClient<MulticastDelegate> cl2(clop2);
    cl2.delegate = &del;

    DefaultMulticastClientOptions clop1 {
        static_sk1,
        {"goldfish"},
        "127.0.0.1:9002"
    };
    DefaultMulticastClient<MulticastDelegate> cl1(clop1);
    cl1.delegate = &del;

    return DefaultMulticastClient<MulticastDelegate>::run_event_loop();
}

Step 4 - Build and run executable

Build using

$ cmake .. -DCMAKE_BUILD_TYPE=Release
$ make

and run using

$ ./tutorial

Building should have worked and the executable should run. You should see subscription related logs printed here indicating a successful connection between both the clients and the Marlin Network. You should also see message logs indicating new messages sent and received using the Marlin Network both here and in the Goldfish node.

Conclusion

In this last part, we learnt how to use the Marlin Multicast SDK to send and receive messages using the Marlin Network.