Node.js的分布式计算框架

Node.js的分布式计算框架

Distributed Computing Framework for Node.js

Early development stage: this project was still under early development, many necessery feature was not done yet, use it on your own risk.

Document

API Reference

A node.js version of Spark, without hadoop or jvm.

You should read tutorial first, then you can learn Spark but use this project instead.

Async API & deferred API

Any api that requires a RDD and generate a result is async, like count, take, max ... Any api that creates a RDD is deferred API, which is not async, so you can chain them like this:

await dcc .parallelize([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) .map(v => v + 1) .filter(v => v % 2 === 0) .take(10); // take is not deferred api but async

Milestones

0.1.x: Basic

local master. rdd & partition creation & release. map & reduce repartition & reduceByKey disk storage partitions cache file loader & saver export module to npm decompresser & compresser use debug module for information/error provide a progress bar. sampler sort object hash(for key) method storage MEMORY_OR_DISK, and use it in sort storage MEMORY_SER，storage in memory but off v8 heap. config default partition count.

0.2.x: Remote mode

distributed master runtime sandbox plugin system remote dependency management aliyun oss loader hdfs loader

How to use

Install from npm(shell only)

npm install -g dcf#oryarn global add dcf

Then you can use command: dcf-shell

Install from npm(as dependency)

npm install --save dcf#oryarn add dcf

Then you can use dcf with javascript or typescript.

Run samples & cli

download this repo, install dependencies

npm install# oryarn

Run samples:

npm run ts-node src/samples/tutorial-0.tsnpm run ts-node src/samples/repartition.ts

Run interactive cli:

npm start

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。