Class: BaseRetailerService

BaseRetailerService(configs)

A Retailer Service Class, it has all the features you need to create your Reatailer Service.

Constructor

new BaseRetailerService(configs)

Create a Retailer Service. Environment variables are highest priority.

For example, you set environment:

expoort GLOBAL_ID=abcd.environment

And in your code, you also set it manually:

new BaseRetailerService({
  GLOBAL_ID:"abcd.manual"
})

So in this case, GLOBAL_ID is abcd.environment, not abcd.manual.

Parameters:
Name Type Description
configs Configurations

Configurations also can set by call baseRetailerService.setConfigs(configs) or set as environment variables
Source:

Members

app :Object

An Express application return by express().

Type:
  • Object
Source:

logger :Object

Before you baseRetailerService.init(), logger is a console, afer you init , logger will be a winston logger

Type:
  • Object
Source:

Methods

connector(customConnectoropt)

Connector response to save data to disk, database or maybe publish to kafka. You can use this to create your own connector. Call this function after init We have json and mongodb two connectors. By default will use json, if you you can use CONNECTOR_TYPE to change connector type. After you call init, will base on CONNECTOR_TYPE to create a connector for you.

Parameters:
Name Type Attributes Description
customConnector Object <optional>

Your custom connector
Source:

express(paramopt) → {BaseRetailerService}

Configure express application

Parameters:
Name Type Attributes Description
param Object <optional>
Properties
Name Type Attributes Default Description
limit string <optional>
 100mb

Controls the maximum request body size. If this is a number, then the value specifies the number of bytes; if it is a string, the value is passed to the bytes library for parsing.
views string | array <optional>

A directory or an array of directories for the application's views. If an array, the views are looked up in the order they occur in the array.
statics string | array <optional>

A directory or an array of directories which to serve static assets, like images, json files and other. You need to pass absolute path. For more detail, please take a look ExpressJS static middleware
Source:
Returns:
Type
BaseRetailerService

generateTask(param) → {Task}

Based on passed url, priority, globalId and metadata generate an task object. You can find task schema from https://docs.bitsky.ai/api/bitsky-restful-api#request-body-array-item-schema

Parameters:
Name Type Description
param Object
Properties
Name Type Attributes Description
url string

web page url that need to be processed
priority integer <optional>

Priority of this task. Only compare priority for same Retailer Service, doesn't compare cross Retailer Service. Bigger value low priority. Priority value 1 is higher than priority value 2
metadata Object <optional>

Additional metadata for this task
suitableProducers Array <optional>

What kind of producers can execute this task
globalId string <optional>

The global id of your Retailer Service. If you didn't pass will get from Configurations.GLOBAL_ID
Source:
Returns:
Type
Task

getConfigs() → {Configurations}

Get Configurations

Source:
Returns:
Type
Configurations

getHomeFolder() → {string}

Get path to the Retailer Home folder

Source:
Returns:
Type
string

(async) getRetailerConfiguration(baseURLopt, globalIdopt) → {Object|Error}

Get retailer configuration by global id. If you didn't pass, then it uses BITSKY_BASE_URL and GLOBAL_ID return by thi.getConfigs()

Parameters:
Name Type Attributes Description
baseURL string <optional>

BitSky Supplier server url
globalId string <optional>

Retailer Configuration Glogbal ID
Source:
Returns:
Type
Object | Error

init()

Init BaseRetailerService, create BaseRetailerService#logger

Source:

(async) listen(portopt)

Start http server and listen to port, also producer start to watch tasks

Parameters:
Name Type Attributes Description
port number <optional>

port number. Default get from Configuration.PORT
Source:

parse(parseFunopt) → {BaseRetailerService}

Get or set parse function

Parameters:
Name Type Attributes Description
parseFun function <optional>

Parse function, it can be function or async function

parseFun will get an object as parameter.

{
  req,
  res
}
  1. req: ExpressJS Request
  2. res: ExpressJS Response

And parseFun need to return a parse result object
Source:
Returns:
Type
BaseRetailerService
Example
const baseRetailerService = require("@bitskyai/retailer-sdk");
const cheerio = require("cheerio");
const parseFun = async function parse({ req, res }) {
  try {
    let collectedTasks = req.body;
    // Tasks that need collected by Producer
    let needCollectTasks = [];
    // Collected data
    let collectedData = [];

    for (let i = 0; i < collectedTasks.length; i++) {
      let item = collectedTasks[i];
      // req.body - https://docs.bitsky.ai/api/bitsky-restful-api#request-body-array-item-schema
      let data = item.dataset.data.content;

      // You can find how to use cheerio from https://cheerio.js.org/
      // cheerio: Fast, flexible & lean implementation of core jQuery designed specifically for the server.
      let $ = cheerio.load(data);

      let targetBaseURL = "http://exampleblog.bitsky.ai/";
      if (item.metadata.type == "bloglist") {
        // get all blogs url in blog list page
        let blogUrls = $("div.post-preview a");
        for (let i = 0; i < blogUrls.length; i++) {
          let $blog = blogUrls[i];
          $blog = $($blog);
          let url = new URL($blog.attr("href"), targetBaseURL).toString();
          needCollectTasks.push(
            baseRetailerService.generateTask({
              url,
              priority: 2,
              metadata: {
                type: "blog",
              },
            })
          );
        }
        let nextUrl = $("ul.pager li.next a").attr("href");
        if (nextUrl) {
          nextUrl = new URL(nextUrl, targetBaseURL).toString();
          needCollectTasks.push(
            baseRetailerService.generateTask({
              url: nextUrl,
              priority: 2,
              metadata: {
                type: "bloglist",
              },
            })
          );
        }
      } else if (item.metadata.type == "blog") {
        collectedData.push({
          title: $("div.post-heading h1").text(),
          author: $("div.post-heading p.meta span.author").text(),
          date: $("div.post-heading p.meta span.date").text(),
          content: $("div.post-container div.post-content").text(),
          url: item.dataset.url,
        });
      } else {
        console.error("unknown type");
      }
    }
    return {
      key: "blogs",
      response: {
        status: 200
      },
      data: collectedData,
      tasks: needCollectTasks,
    };
  } catch (err) {
    console.log(`parse error: ${err.message}`);
  }
};
baseRetailerService.parse(parseFun)

routers(paramopt)

Configure express router

Parameters:
Name Type Attributes Description
param Object <optional>
Properties
Name Type Attributes Description
skipRouters object <optional>

which router you want to skip, when you skip a router, then you need implement by yourself, otherwise it maybe cause issue, especially for tasks

Properties
Name Type Attributes Default Description
index boolean <optional>
 false

skip index router
health boolean <optional>
 false

skip health router
tasks boolean <optional>
 false

skip tasks router
indexOptions IndexOptions <optional>

Data you want to overwrite default index data
Source:

sendTasksToSupplier(tasks) → {Promise}

Add tasks to BitSky application

Parameters:
Name Type Description
tasks array

Array of Task want to be added
Source:
Returns:
Type
Promise

setConfigs(configs)

Set BaseRetailerService configurations

Parameters:
Name Type Description
configs Configurations

Configuraions you want to set
Source:

(async) stop()

Destory this retailer service

Source:

trigger(triggerFunopt) → {BaseRetailerService}

Get or set trigger function

Parameters:
Name Type Attributes Description
triggerFun function <optional>

Trigger function, it can be function or async function

triggerFun will get an object as parameter.

{
  req,
  res
}
  1. req: ExpressJS Request
  2. res: ExpressJS Response

And triggerFun need to return a trigger result object
Source:
Returns:
Type
BaseRetailerService
Example
const baseRetailerService = require("@bitskyai/retailer-sdk");
const triggerFun = async function trigger({ req, res }) {
  return {
    tasks: [
      baseRetailerService.generateTask({
        url: "http://exampleblog.bitsky.ai/",
        priority: 1,
        metadata: { type: "bloglist" },
      }),
    ],
  };
};
baseRetailerService.trigger(triggerFun)