huey：小型多线程任务队列

huey：小型多线程任务队列

a lightweight alternative.

huey is:

a task queue (2019-04-01: version 2.0 released)written in python (2.7+, 3.4+)clean and simple APIredis, sqlite, or in-memory storageexample code.

huey supports:

multi-process, multi-thread or greenlet task execution modelsschedule tasks to execute at a given time, or after a given delayschedule recurring tasks, like a crontabautomatically retry tasks that failtask prioritizationtask result storagetask lockingtask pipelines and chains

At a glance

from huey import RedisHuey, crontabhuey = RedisHuey('my-app', host='redis.myapp.com')@huey.task()def add_numbers(a, b): return a + b@huey.task(retries=2, retry_delay=60)def flaky_task(url): # This task might fail, in which case it will be retried up to 2 times # with a delay of 60s between retries. return this_might_fail(url)@huey.periodic_task(crontab(minute='0', hour='3'))def nightly_backup(): sync_all_data()

Calling a task-decorated function will enqueue the function call for execution by the consumer. A special result handle is returned immediately, which can be used to fetch the result once the task is finished:

>>> from demo import add_numbers>>> res = add_numbers(1, 2)>>> res>>> res()3

Tasks can be scheduled to run in the future:

>>> res = add_numbers.schedule((2, 3), delay=10) # Will be run in ~10s.>>> res(blocking=True) # Will block until task finishes, in ~10s.5

For much more, check out the guide or take a look at the example code.

Running the consumer

Run the consumer with four worker processes:

$ huey_consumer.py my_app.huey -k process -w 4

To run the consumer with a single worker thread (default):

$ huey_consumer.py my_app.huey

If your work-loads are mostly IO-bound, you can run the consumer with threads or greenlets instead. Because greenlets are so lightweight, you can run quite a few of them efficiently:

$ huey_consumer.py my_app.huey -k greenlet -w 32

Storage

Huey's design and feature-set were informed by the capabilities of the Redis database. Redis is a fantastic fit for a lightweight task queueing library like Huey: it's self-contained, versatile, and can be a multi-purpose solution for other web-application tasks like caching, event publishing, analytics, rate-limiting, and more.

Although Huey was designed with Redis in mind, the storage system implements a simple API and many other tools could be used instead of Redis if that's your preference.

Huey comes with builtin support for Redis, Sqlite and in-memory storage.

Documentation

See Huey documentation.

Project page

See source code and issue tracker on Github.

Huey is named in honor of my cat:

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。