EightFish

What is EightFish?

EightFish is a web MVC framework to develop decentralized applications.

Deeply, EightFish is a development framework (maybe the first one) for the Open Data Application (ODA), implementing the Open Data Application Model (ODAM). You can think of ODA is a type of data-kind decentralized application of Web3. The theory of ODA and ODAM is located here. In a short description: EightFish powers ODAs, ODAs constitute the OpenWeb, which is a subset of Web3.

EightFish makes devs to develop a decentralized application in Web2 development style, rather than the smart contract style. Unlike the smart contract blockchain tech stack most DApps adopt, EightFish makes your own network, a sovereign network which doesn't rely on any other Web3 layers or services.

By some elaberate designs, EightFish reaches the experiences of Web2/Internet web development, but for the OpenWeb/Web3 decentralized application.

NOTICE: EightFish itself is not a service/platform/serverless/layer, it is just a dev framework tool.

License

GPLv3.0

Features

EightFish has following highlights:

High performance. EightFish put user experience as the first level target, it has been optimized on all aspects to achieve this taget.
Low latency. EightFish makes it to get consensus result in upto 2 seconds.
Fully on-chain actions. EightFish use blockchain to keep the openness of the data mandatory, all actions are related to on-chain storage or checking.
Powerful off-chain abilities. EightFish has a full-fledged off-chain worker outside of the blockchain node, it can perform general computation, although some limitations applied.
Powerful query ability. EightFish grants you the ability to use query statement of SQL out of box, which is far beyond the schemes supplied by current mainstream Web3 tech stack (e.g. Eth + The Graph). The query result will be returned in dozens of ms, after the internal validation.
An easy Web CRUD interface for programmer. A traditional Web2 (or Internet) programmer would feel home at writing the code of an EightFish application, it's the same.
All in WebAssembly. All components are written in Rust, and compiled to WebAssembly to distributed and executed.

In addtion, an EightFish App (or Open Data App) has following extra features:

An EightFish App will have a sovereign network for it.
No token (asset) on this app platform/network, thus NO xFi for the data on this platform, and NO token incentives for validators;
An EightFish App is just (or better as) a layer of low level data service, with no UI, no static asset files, no higher business logic in it. It needs some form of front services connected to it to provide direct business products (data aggregations, Web UI, Phone UI, etc.) for the end user;
One EightFish App/network holds one application protocol, one application protocol can hold multiple products for users. Different products are connected to the same backend data service, share the same dataset among all products.
An EightFish application/product supports both the Web2 style account via Oauth, and the Web3 style account via cryptographic algorithm.
On the side of user experience, if not told, you can barely recognize it as a Web3 application. Too good to ignore.

How to Get Started

Create a new project

From the template project of ef_example_simple_standalone.

git clone https://github.com/eightfish-org/ef_example_simple_standalone

Modify this template and git config infomation.

Compile this repository

cd ef_example_simple_standalone
spin build

here we assumed that you have installed the spin binary and the Rust toolchain set. If you didn't, do it by:

# install rust at first, and add the following component
rustup target add wasm32-wasi

# download spin v1.3.0
cd /tmp 
curl -fsSL https://developer.fermyon.com/downloads/install.sh | bash -s -- -v v1.3.0
mv /tmp/spin ~/.cargo/bin/

Copy the spin binary to current directory

cp ~/.cargo/bin/spin .

Build the app docker

./build_app.sh

Run docker compose.

docker compose -f docker-compose-1node.yml up

after a while,

Test

We use hurl as the client to do testing. You can install it by:

cargo install hurl

And then,

cd flow_tests

# create new artile row
hurl new_article.hurl

# it returns something like:
# {"result":"Ok","id":"5wzxHoJnQd5QhbGcdKkesGiEwtUkynPY4JFrUrm9Us5q"}

# copy the returned id and paste to the right place of the next command line to get this article
hurl --variable id=5wzxHoJnQd5QhbGcdKkesGiEwtUkynPY4JFrUrm9Us5q get_one_article.hurl

# it returns something like:
# [{"id":"5wzxHoJnQd5QhbGcdKkesGiEwtUkynPY4JFrUrm9Us5q","title":"test111","content":"this is the content of test111","authorname":"mike tang"}]

Tech Stack

EightFish is written in Rust overall. Thanks to the following technologies:

Substrate. The subnode component is built using Substrate, the most powerful blockchain framework made by Parity.
Subxt. The subxtproxy is built using Subxt, the client RPC library of Substrate.
Spin framework. An innovative webassembly framework for the future of the cloud.
redis. No explanation.
postgres db. No explanation.

Substrate

The role of Substrate in EightFish.

Used to record the incoming raw writing requests, bake them into blocks, like a log system
Used to sync runtime state among all EightFish nodes (Substrate network), further to coordinate the state of the SQL db among all nodes
Used to store the version of wasm code and make the code of spin worker upgrade forklessly
(not sure) Used to interoperate with other Substrate-based chains by leveraging existing pallets

There is a great slide to explain EightFish for Substrate developers.

All in Wasm

Almost all EightFish components are compiled into webassembly to run, except for redis and postgres.

We believe that the wasm bytecode will be the standard format of software distribution in the future.

Examples

There are some examples you can get to start on.

Anatomy

The simplest interface usually means the most complex innternal mechanism. Here we give the insights of the EightFish framework.

Core Mapping

This picture illustrates the core mapping theory between off-chain sql db and the on-chain runtime storage.

Arch of the EightFish Node

Explanation:

There are some components in an EightFish node.

subnode: the blockchain node located inside of an EightFish node.
subxtproxy: the subnode rpc client used to connect the subnode with the spin worker.
redis: used as msg channels and data caches
postgres: used as the storage of raw data
http gate: used as the interface of http data service
spin worker: acts as the core business engine in EightFish workflow.
MVC: an easy engineering layer for programmer to write logic, in a style of Web CRUD.
Tiny ORM: a set of helper ORM utilities to make the interactions with SQL statements easier and safer.

Workflows

The followings are sequence diagrams for internal EightFish.

Post (update) workflow

Query (read) workflow

MVC Framework

EightFish embeds a thin MVC layer into it to make the programming life easier. If you come from the traditional Web developments, familiar with frameworks like Spring, Django, Flask, Express, Rocket and others, you will feel home when use EightFish.

In this page, we will use the Simple Example to demostrate.

App Entry

An EightFish app is actually a Spin redis-triggered application. The entry of this type of application looks like:

#[redis_component]
fn on_message(message: Bytes) -> Result<()> { }

You can look into the lib.rs file to learn the style.

The on_message function is just a boilerplate to comply with. In its body, we need to build the instance of our app, and mount it to spin_worker::Worker, and call worker.work(message) to process this incoming message.

Every time the message comes (from the redis channel), the EightFish App handler will process this message, and give response to somewhere (some caches, which is defined by components spin_worker and http_gate together, not the channel message comes from) in redis.

App Instance

Every EightFish App should create an EightFish App instance, like:

pub fn build_app() -> EightFishApp {
    let mut sapp = EightFishApp::new();
    sapp.add_global_filter(Box::new(MyGlobalFilter))
        .add_module(Box::new(article::ArticleModule));

    sapp
}

The above function name is arbitrary, but should return the type of EightFishApp. In its body we create the instance of EightFishApp, mount global filter of the app and all modules implementing the application logic to this instance.

Global Filter

The global filter of EightFish is for all incoming requests. Some actions (like cookie verifying, authentication, etc.) should be checked before any specific logic being executed. So we need this mechanism to tackle them.

impl GlobalFilter for MyGlobalFilter {
    fn before(&self, _req: &mut Request) -> EightFishResult<()> {
        Ok(())
    }

    fn after(&self, _req: &Request, _res: &mut Response) -> EightFishResult<()> {
        Ok(())
    }
}

GlobalFilter is a trait defined by EightFish SDK. And there are two methods in it: before() and after(). The before() is used to process request ahead of any other biz logic code, and the after() is used to process the response after all biz logic but before responding to user, it's the last step we can write something to attach or modify information to the response.

Modules

Every specific biz logic should be put into each specific module, and then mount these modules into the app instance described above. You can look into the article.rs file to get a concrete picture of the structure of a module.

First, you need to define a struct type, like:

pub struct ArticleModule;

and implement the Module trait on it, to fill the router with some url endpoints:

impl Module for ArticleModule {
    fn router(&self, router: &mut Router) -> Result<()> {
        router.get("/article/:id", Self::get_one);
        router.post("/article/new", Self::new_article);
        router.post("/article/update", Self::update);
        router.post("/article/delete/:id", Self::delete);

        Ok(())
    }
}

The above snippet fills the url router Router by implementing the fn router, the router supports two kinds of methods: get and post, correspond to the HTTP GET and POST methods respectively.

Besides that, in the trait Module, we can implement the second level filters: before() and after(). These two filters apply on the url matches within this local module, not affecting other urls in other modules. So it is a kind of filter in module level. The EightFish::Module is defined as follow:

pub trait EightFishModule: Sync + Send {
    /// module before filter, will be executed before handler
    fn before(&self, _req: &mut EightFishRequest) -> Result<()> {
        Ok(())
    }

    /// module after filter, will be executed after handler
    fn after(&self, _req: &EightFishRequest, _res: &mut EightFishResponse) -> Result<()> {
        Ok(())
    }

    /// module router method, used to write router collection of this module here
    fn router(&self, router: &mut EightFishRouter) -> Result<()>;
}

URL Handler

The corresponding handlers are implemented onto the module struct, as a method of this struct, like:

fn handler_name(req: &mut Request) -> Result<Response> {

Every url handler has unified function signature: a req: &mut Request as parameter, and a Result<Response> as function returning.

In the handler function, you can process the request at the third level.

Middleware Mechanism

In EightFish, middleware is just a normal function. It accepts the req: &mut Request and return Result<()>.

pub fn middleware_fn(req: &mut Request) -> Result<()> {

The middleware function could be placed in the global filter, module filter, or every handler function. This conduct a flexible three levels of middleware system.

Initial Globals

There is a method on EightFishApp: init_global(). You can put the global variables which should exist as long as the EightFish app's lifecycle in it. This is a mechanism for shared data between different requests.

In the provided closure, you need to insert your desired data into the extension part of the request, as follows:


let a_global = Arc::new(Mutex::new(..))

...
app.init_global(|req: &mut Request| -> Result<()> {
	req.ext_mut().insert("a_global", a_global);
})
...

Helper Macros

EightFish designs some helper macros to improve the experience of logic coding, especially on interacting with SQL db.

EightFish provides a derived macro named EightFishModel for user's model.

For example, you just need to put this macro into the derive section above a model (struct) definition.

#[derive(Debug, Clone, Serialize, Deserialize, EightFishModel, Default)]
pub struct Article {
    id: String,
    title: String,
    content: String,
    authorname: String,
}

After that, the struct Article will gain some powerful functionalities like:

fn model_name() -> String {}
fn field_names() -> String {}
fn build_insert_sql() -> String {}
fn from_row(row: Vec<DbValue>) -> #ident {}
...

These functionalities make you write biz code easily, rapidly and happily.

You can refer to the detailed inline doc here.

Run it

After all, switch into the app directory, and type:

spin up

to run this app up.

On-chain Service

EightFish is used to develop the decentralized application, so there are some important differences compared to traditional web framework.

For example, in EightFish app, we couldn't use the random function from your local environment, we MUST use the random data from the on-chain runtime. This will affect the generation of all models' id.

Random string

EightFish offers a random string source out of box, this random string will change at every request. You can get the inner on-chain random string by:

let id = req
    .ext()
    .get("random_str")
    .ok_or(anyhow!("id error"))?
    .to_owned();

Please refer Simple Example to get the context of above code.

Timestamp

The timestamp has the same case with random string, we should use on-chain timestamp rather than local timestamp facility.

You can similarily retrieve the on-chain time by:

let time = req
    .ext()
    .get("time")
    .ok_or(anyhow!("generate time failed"))?
    .parse::<i64>()?;

The unit of the on-chain timestamp is micro second.

Others

Generally speaking, you can't do any actions which would violate the state consensus with other EightFish nodes.

TinyORM

EightFish offers a set of basic ORM helper functions to make life easier when interacting with SQL.

CRUD

Create

You can use the method of instance.build_insert() to get an insert SQL statement from a model. For example:

let article = Article {
    id,
    title,
    content,
    authorname,
};

let (sql_statement, sql_params) = article.build_insert();
_ = pg::execute(&pg_addr, &sql_statement, &sql_params)?;