Beanstalkd

What is an asynchronous queue

The classic wikipedia quote (Message queue)

In computer science, message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content. Group communication systems provide similar kinds of functionality.

So one part of a system puts a message into a queue for another part to read from, and then act upon. The asynchronous nature means that each side is otherwise independent from the other, and does not wait for a response. That independence is an important part of the nature of the system though – and we’ll see later how some of the more advanced functionality for our software of choice here can give some extraordinary flexibility to what can be done.

Why use a queuing system?

You’d be surprised how few things need to happen right now – you go and buy a fancy coffee, and they write your order down, and put it into the queue for the Barista to make it. That disconnected set of actions works exceeding well for such distributed system (see Starbucks Does Not Use Two-Phase Commit)

In much the same way as you not getting your coffee till it’s made, what about web-sites that have to fetch (or produce) information. A couple of the simpler examples are when you’ve uploaded an image onto Flickr.com. That image has to be stored, and then resized into several files. If it’s a large image though, it would take some time, and a lot of resources to be able to do that while you waited – time that you’re left twiddling your thumbs. Instead, it returns immediately, and tells you that the image is being handled in the background – and in a few seconds, or maybe minutes, it shows up on your page.

How about waiting a few seconds for other information? How about, when you login to a social media website, it returns a simple webpage immediately with what it’s got to hand, but then in the background, checks how many new messages you have, and displays them either by updating the page (with ajax), or when you view a different page. Is it so vital you find out that you have thirty old messages, and a few new ones – right now? For a web-mail system like Gmail, or Yahoo Mail, that is the point – but what about on another kind of site?

Beanstalkd

---

Beanstalkd was first developed to solve the needs of a popular web application (Causes on Facebook). Currently, it is an absolutely reliable, easy to install messaging service which is perfect to get started with and use.

As mentioned earlier, Beanstalkd's main use case is to manage the workflow between different parts and workers of your application deployment stack through work queues and messages, similar to other popular solutions such as RabbitMQ. However, the way Beanstalkd is created to work sets it apart from the rest.

Since its inception, unlike other solutions, Beanstalkd was intended to be a work queue and not an umbrella tool to cover many needs. To achieve this purpose, it was built as a lightweight and rapidly functioning application based on C programming language. Its lean architecture also allows it to be installed and used very simply, making it perfect for a majority of use cases.

Features

---

Being able to monitor jobs with a returned ID, returned upon creation, is only one of the features of Beanstalkd that sets it apart from the rest. Some other interesting features offered are:

• Persistence - Beanstalkd operates in-memory but offers persistence support as well.
• Prioritisation - unlike most alternatives, Beanstalkd offers prioritisation for different tasks to handle urgent things when they are needed to.
• Distribution - different server instances can be distributed similarly to how Memcached works.
• Burying - it is possible to indefinitely postpone a job (i.e. a task) by burying it.
• Third party tools - Beanstalkd comes with a variety of third-party tools including CLIs and web-based management consoles.
• Expiry - jobs can be set to expire and auto-queue later (TTR - Time To Run).

Beanstalkd Use-case Examples

---

Some exemplary use-cases for Banstalkd are:

• Allowing web servers to respond to requests quickly instead of being forced to perform resource-heavy procedures on the spot
• Performing certain jobs at certain intervals (i.e. crawling the web)
• Distributing a job to multiple workers for processing
• Letting offline clients (e.g. a disconnected user) fetch data at a later time instead of having it lost permanently through a worker
• Introducing fully asynchronous functionality to the backend systems
• Ordering and prioritising tasks
• Balancing application load between different workers
• Greatly increase reliability and uptime of your application
• Processing CPU intensive jobs (videos, images etc.) later
• Sending e-mails to your lists
• and more.

Beanstalkd Elements

---

Just like most applications, Beanstalkd comes with its own jargon to explain its parts.

Tubes / Queues

---

Beanstalkd Tubes translate to queues from other messaging applications. They are through where jobs (or messages) are transferred to consumers (i.e. workers).

Jobs / Messages

---

Since Beanstalkd is a "work queue", what's transferred through tubes are referred as jobs - which are similar to messages being sent.

Producers / Senders

---

Producers, similar to Advanced Message Queuing Protocol's definition, are applications which create and send a job (or a message). They are to be used by the consumers.

Consumers / Receivers

---

Receivers are different applications of the stack which get a job from the tube, created by a producer for processing.

http://alister.github.io/presentations/Beanstalkd/

Install

yum install -y beanstalkd

Using Beanstalkd


Upon installing, you can start working with the Beanstalkd server. Here are the options for running the daemon:
 -b DIR   wal directory
 -f MS    fsync at most once every MS milliseconds (use -f0 for "always fsync")
 -F       never fsync (default)
 -l ADDR  listen on address (default is 0.0.0.0)
 -p PORT  listen on port (default is 11300)
 -u USER  become user and group
 -z BYTES set the maximum job size in bytes (default is 65535)
 -s BYTES set the size of each wal file (default is 10485760)
            (will be rounded up to a multiple of 512 bytes)
 -c       compact the binlog (default)
 -n       do not compact the binlog
 -v       show version information
 -V       increase verbosity
 -h       show this help


Example Usage:

# Usage: beanstalkd -l [ip address] -p [port #]
# For local only access:
beanstalkd -l 127.0.0.1 -p 11301 &


Managing The Service:


If installed through the package manager (i.e. aptitude), you will be able to manage the Beanstalkd daemon as a service.
# To start the service:
service beanstalkd start

# To stop the service:
service beanstalkd stop

# To restart the service:
service beanstalkd restart

# To check the status:
service beanstalkd status

PHP is often considered the glue of the Internet because it’s can be used for a lot of different things. There are so many built in functions and features that you are unlikely to use close to all of them if you work with PHP the rest of your life. This is great because we can solve various problems in our web apps without resorting to bringing in another language. Sometimes though, our application logic can get quite large or resource intensive. This translates into slow response times for the users of our apps if certain operations are tied into user events.

Let’s say we’re running dating site. Users are asked to enter a bunch of information at sign-up. This information will then be used by our application to match the new user with existing users and then email or send them a message with their new matches. This can potentially be a lengthy operation. If we run this as soon as the user hits submit they could be greeted by a long wait for the page or even worse a timeout error. To prevent this we can use a queue.

In this example we’ll be using a simple messaging queue called Beanstalkd and a PHP wrapper package called Pheanstalk to interact with it.

First install Beanstalkd using whatever is the norm in your current linux distribution whether that is yum or apt-get. You can also clone the github repo and compile from source. Once installed you can run Beanstalkd locally from the command line.









1



$ beanstalkd -l 127.0.0.1








Next let’s pull in Pheanstalk. Create a composer.json file.









1

2

3

4

5



{

    "require": {

        "pda/pheanstalk": "2.1.0"

    }

}








Run ‘composer update’. Now let’s create a file called ‘functions.php’ to hold our super intensive matching function. This will be used by our worker script.









1

2

3

4

5

6



<?php



function find_matches($data) {

    // Run super awesome calculations

    // Send user our email

}








Now let’s place a job on the queue. We’ll call this file ‘find_matches.php’.









1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18



<?php



include 'vendor/autoload.php';



$queue = new Pheanstalk_Pheanstalk('127.0.0.1');



$job = new stdClass();

// the function to run

$job->function = 'find_matches';

// our user entered data

$job->user_data = $_POST;



// beanstalkd uses strings so we json_encode our job for storage

$job_data = json_encode($job);



// place our job into the queue into a tube we'll call matching

$queue->useTube('matching')

    ->put($job_data);








Now our job is in the queue waiting for a worker to come along and grab it. Let’s create one in ‘worker.php’.









1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24



<?php

include 'vendor/autoload.php';

include 'functions.php';



$queue = new Pheanstalk_Pheanstalk('127.0.0.1');



while (1) {

    // grab the next job off the queue and reserve it

    $job = $queue->watch('matching')

        ->ignore('default')

        ->reserve();

    

    // decode the json data

    $job_data = json_decode($job->getData(), false);



    $function = $job_data->function;

    $data = $job_data->user_data;

    

    // run the function

    $function($data);



    // remove the job from the queue

    $queue->delete($job);

}








Now the worker script will not run automatically so you will need to either run it as a cron job or use a tool like daemontools to keep it running all the time in the background.

This is just a simple example but hopefully it gives you an idea of what you can accomplish with queues.

= Beanstalk Protocol =



Protocol

--------



The beanstalk protocol runs over TCP using ASCII encoding. Clients connect,

send commands and data, wait for responses, and close the connection. For each

connection, the server processes commands serially in the order in which they

were received and sends responses in the same order. All integers in the

protocol are formatted in decimal and (unless otherwise indicated)

nonnegative.



Names, in this protocol, are ASCII strings. They may contain letters (A-Z and

a-z), numerals (0-9), hyphen ("-"), plus ("+"), slash ("/"), semicolon (";"),

dot ("."), dollar-sign ("$"), and parentheses ("(" and ")"), but they may not

begin with a hyphen. They are terminated by white space (either a space char or

end of line). Each name must be at least one character long.



The protocol contains two kinds of data: text lines and unstructured chunks of

data. Text lines are used for client commands and server responses. Chunks are

used to transfer job bodies and stats information. Each job body is an opaque

sequence of bytes. The server never inspects or modifies a job body and always

sends it back in its original form. It is up to the clients to agree on a

meaningful interpretation of job bodies.



There is no command to close the connection -- the client may simply close the

TCP connection when it no longer has use for the server. However, beanstalkd

performs very well with a large number of open connections, so it is usually

better for the client to keep its connection open and reuse it as much as

possible. This also avoids the overhead of establishing new TCP connections.



If a client violates the protocol (such as by sending a request that is not

well-formed or a command that does not exist) or if the server has an error,

the server will reply with one of the following error messages:



 - "OUT_OF_MEMORY\r\n" The server cannot allocate enough memory for the job.

   The client should try again later.



 - "INTERNAL_ERROR\r\n" This indicates a bug in the server. It should never

   happen. If it does happen, please report it at

   http://groups.google.com/group/beanstalk-talk.



 - "DRAINING\r\n" This means that the server has been put into "drain mode"

   and is no longer accepting new jobs. The client should try another server

   or disconnect and try again later.



 - "BAD_FORMAT\r\n" The client sent a command line that was not well-formed.

   This can happen if the line does not end with \r\n, if non-numeric

   characters occur where an integer is expected, if the wrong number of

   arguments are present, or if the command line is mal-formed in any other

   way.



 - "UNKNOWN_COMMAND\r\n" The client sent a command that the server does not

   know.



These error responses will not be listed in this document for individual

commands in the following sections, but they are implicitly included in the

description of all commands. Clients should be prepared to receive an error

response after any command.



As a last resort, if the server has a serious error that prevents it from

continuing service to the current client, the server will close the

connection.



Job Lifecycle

-------------



A job in beanstalk gets created by a client with the "put" command. During its

life it can be in one of four states: "ready", "reserved", "delayed", or

"buried". After the put command, a job typically starts out ready. It waits in

the ready queue until a worker comes along and runs the "reserve" command. If

this job is next in the queue, it will be reserved for the worker. The worker

will execute the job; when it is finished the worker will send a "delete"

command to delete the job.



Here is a picture of the typical job lifecycle:





   put            reserve               delete

  -----> [READY] ---------> [RESERVED] --------> *poof*







Here is a picture with more possibilities:







   put with delay               release with delay

  ----------------> [DELAYED] <------------.

                        |                   |

                        | (time passes)     |

                        |                   |

   put                  v     reserve       |       delete

  -----------------> [READY] ---------> [RESERVED] --------> *poof*

                       ^  ^                |  |

                       |   \  release      |  |

                       |    `-------------'   |

                       |                      |

                       | kick                 |

                       |                      |

                       |       bury           |

                    [BURIED] <---------------'

                       |

                       |  delete

                        `--------> *poof*





The system has one or more tubes. Each tube consists of a ready queue and a

delay queue. Each job spends its entire life in one tube. Consumers can show

interest in tubes by sending the "watch" command; they can show disinterest by

sending the "ignore" command. This set of interesting tubes is said to be a

consumer's "watch list". When a client reserves a job, it may come from any of

the tubes in its watch list.



When a client connects, its watch list is initially just the tube named

"default". If it submits jobs without having sent a "use" command, they will

live in the tube named "default".



Tubes are created on demand whenever they are referenced. If a tube is empty

(that is, it contains no ready, delayed, or buried jobs) and no client refers

to it, it will be deleted.



Producer Commands

-----------------



The "put" command is for any process that wants to insert a job into the queue.

It comprises a command line followed by the job body:



put <pri> <delay> <ttr> <bytes>\r\n

<data>\r\n



It inserts a job into the client's currently used tube (see the "use" command

below).



 - <pri> is an integer < 2**32. Jobs with smaller priority values will be

   scheduled before jobs with larger priorities. The most urgent priority is 0;

   the least urgent priority is 4294967295.



 - <delay> is an integer number of seconds to wait before putting the job in

   the ready queue. The job will be in the "delayed" state during this time.



 - <ttr> -- time to run -- is an integer number of seconds to allow a worker

   to run this job. This time is counted from the moment a worker reserves

   this job. If the worker does not delete, release, or bury the job within

   <ttr> seconds, the job will time out and the server will release the job.

   The minimum ttr is 1. If the client sends 0, the server will silently

   increase the ttr to 1.



 - <bytes> is an integer indicating the size of the job body, not including the

   trailing "\r\n". This value must be less than max-job-size (default: 2**16).



 - <data> is the job body -- a sequence of bytes of length <bytes> from the

   previous line.



After sending the command line and body, the client waits for a reply, which

may be:



 - "INSERTED <id>\r\n" to indicate success.



   - <id> is the integer id of the new job



 - "BURIED <id>\r\n" if the server ran out of memory trying to grow the

   priority queue data structure.



   - <id> is the integer id of the new job



 - "EXPECTED_CRLF\r\n" The job body must be followed by a CR-LF pair, that is,

   "\r\n". These two bytes are not counted in the job size given by the client

   in the put command line.



 - "JOB_TOO_BIG\r\n" The client has requested to put a job with a body larger

   than max-job-size bytes.



The "use" command is for producers. Subsequent put commands will put jobs into

the tube specified by this command. If no use command has been issued, jobs

will be put into the tube named "default".



use <tube>\r\n



 - <tube> is a name at most 200 bytes. It specifies the tube to use. If the

   tube does not exist, it will be created.



The only reply is:



USING <tube>\r\n



 - <tube> is the name of the tube now being used.



Worker Commands

---------------



A process that wants to consume jobs from the queue uses "reserve", "delete",

"release", and "bury". The first worker command, "reserve", looks like this:



reserve\r\n



Alternatively, you can specify a timeout as follows:



reserve-with-timeout <seconds>\r\n



This will return a newly-reserved job. If no job is available to be reserved,

beanstalkd will wait to send a response until one becomes available. Once a

job is reserved for the client, the client has limited time to run (TTR) the

job before the job times out. When the job times out, the server will put the

job back into the ready queue. Both the TTR and the actual time left can be

found in response to the stats-job command.



A timeout value of 0 will cause the server to immediately return either a

response or TIMED_OUT.  A positive value of timeout will limit the amount of

time the client will block on the reserve request until a job becomes

available.



During the TTR of a reserved job, the last second is kept by the server as a

safety margin, during which the client will not be made to wait for another

job. If the client issues a reserve command during the safety margin, or if

the safety margin arrives while the client is waiting on a reserve command,

the server will respond with:



DEADLINE_SOON\r\n



This gives the client a chance to delete or release its reserved job before

the server automatically releases it.



TIMED_OUT\r\n



If a non-negative timeout was specified and the timeout exceeded before a job

became available, the server will respond with TIMED_OUT.



Otherwise, the only other response to this command is a successful reservation

in the form of a text line followed by the job body:



RESERVED <id> <bytes>\r\n

<data>\r\n



 - <id> is the job id -- an integer unique to this job in this instance of

   beanstalkd.



 - <bytes> is an integer indicating the size of the job body, not including

   the trailing "\r\n".



 - <data> is the job body -- a sequence of bytes of length <bytes> from the

   previous line. This is a verbatim copy of the bytes that were originally

   sent to the server in the put command for this job.



The delete command removes a job from the server entirely. It is normally used

by the client when the job has successfully run to completion. A client can

delete jobs that it has reserved, ready jobs, and jobs that are buried. The

delete command looks like this:



delete <id>\r\n



 - <id> is the job id to delete.



The client then waits for one line of response, which may be:



 - "DELETED\r\n" to indicate success.



 - "NOT_FOUND\r\n" if the job does not exist or is not either reserved by the

   client, ready, or buried. This could happen if the job timed out before the

   client sent the delete command.



The release command puts a reserved job back into the ready queue (and marks

its state as "ready") to be run by any client. It is normally used when the job

fails because of a transitory error. It looks like this:



release <id> <pri> <delay>\r\n



 - <id> is the job id to release.



 - <pri> is a new priority to assign to the job.



 - <delay> is an integer number of seconds to wait before putting the job in

   the ready queue. The job will be in the "delayed" state during this time.



The client expects one line of response, which may be:



 - "RELEASED\r\n" to indicate success.



 - "BURIED\r\n" if the server ran out of memory trying to grow the priority

   queue data structure.



 - "NOT_FOUND\r\n" if the job does not exist or is not reserved by the client.



The bury command puts a job into the "buried" state. Buried jobs are put into a

FIFO linked list and will not be touched by the server again until a client

kicks them with the "kick" command.



The bury command looks like this:



bury <id> <pri>\r\n



 - <id> is the job id to release.



 - <pri> is a new priority to assign to the job.



There are two possible responses:



 - "BURIED\r\n" to indicate success.



 - "NOT_FOUND\r\n" if the job does not exist or is not reserved by the client.



The "touch" command allows a worker to request more time to work on a job.

This is useful for jobs that potentially take a long time, but you still want

the benefits of a TTR pulling a job away from an unresponsive worker.  A worker

may periodically tell the server that it's still alive and processing a job

(e.g. it may do this on DEADLINE_SOON).



The touch command looks like this:



touch <id>\r\n



 - <id> is the ID of a job reserved by the current connection.



There are two possible responses:



 - "TOUCHED\r\n" to indicate success.



 - "NOT_FOUND\r\n" if the job does not exist or is not reserved by the client.



The "watch" command adds the named tube to the watch list for the current

connection. A reserve command will take a job from any of the tubes in the

watch list. For each new connection, the watch list initially consists of one

tube, named "default".



watch <tube>\r\n



 - <tube> is a name at most 200 bytes. It specifies a tube to add to the watch

   list. If the tube doesn't exist, it will be created.



The reply is:



WATCHING <count>\r\n



 - <count> is the integer number of tubes currently in the watch list.



The "ignore" command is for consumers. It removes the named tube from the

watch list for the current connection.



ignore <tube>\r\n



The reply is one of:



 - "WATCHING <count>\r\n" to indicate success.



   - <count> is the integer number of tubes currently in the watch list.



 - "NOT_IGNORED\r\n" if the client attempts to ignore the only tube in its

   watch list.



Other Commands

--------------



The peek commands let the client inspect a job in the system. There are four

variations. All but the first operate only on the currently used tube.



 - "peek <id>\r\n" - return job <id>.



 - "peek-ready\r\n" - return the next ready job.



 - "peek-delayed\r\n" - return the delayed job with the shortest delay left.



 - "peek-buried\r\n" - return the next job in the list of buried jobs.



There are two possible responses, either a single line:



 - "NOT_FOUND\r\n" if the requested job doesn't exist or there are no jobs in

   the requested state.



Or a line followed by a chunk of data, if the command was successful:



FOUND <id> <bytes>\r\n

<data>\r\n



 - <id> is the job id.



 - <bytes> is an integer indicating the size of the job body, not including

   the trailing "\r\n".



 - <data> is the job body -- a sequence of bytes of length <bytes> from the

   previous line.



The kick command applies only to the currently used tube. It moves jobs into

the ready queue. If there are any buried jobs, it will only kick buried jobs.

Otherwise it will kick delayed jobs. It looks like:



kick <bound>\r\n



 - <bound> is an integer upper bound on the number of jobs to kick. The server

   will kick no more than <bound> jobs.



The response is of the form:



KICKED <count>\r\n



 - <count> is an integer indicating the number of jobs actually kicked.



The stats-job command gives statistical information about the specified job if

it exists. Its form is:



stats-job <id>\r\n



 - <id> is a job id.



The response is one of:



 - "NOT_FOUND\r\n" if the job does not exist.



 - "OK <bytes>\r\n<data>\r\n"



   - <bytes> is the size of the following data section in bytes.



   - <data> is a sequence of bytes of length <bytes> from the previous line. It

     is a YAML file with statistical information represented a dictionary.



The stats-job data is a YAML file representing a single dictionary of strings

to scalars. It contains these keys:



 - "id" is the job id



 - "tube" is the name of the tube that contains this job



 - "state" is "ready" or "delayed" or "reserved" or "buried"



 - "pri" is the priority value set by the put, release, or bury commands.



 - "age" is the time in seconds since the put command that created this job.



 - "time-left" is the number of seconds left until the server puts this job

   into the ready queue. This number is only meaningful if the job is

   reserved or delayed. If the job is reserved and this amount of time

   elapses before its state changes, it is considered to have timed out.



 - "timeouts" is the number of times this job has timed out during a

   reservation.



 - "releases" is the number of times a client has released this job from a

   reservation.



 - "buries" is the number of times this job has been buried.



 - "kicks" is the number of times this job has been kicked.



The stats-tube command gives statistical information about the specified tube

if it exists. Its form is:



stats-tube <tube>\r\n



 - <tube> is a name at most 200 bytes. Stats will be returned for this tube.



The response is one of:



 - "NOT_FOUND\r\n" if the tube does not exist.



 - "OK <bytes>\r\n<data>\r\n"



   - <bytes> is the size of the following data section in bytes.



   - <data> is a sequence of bytes of length <bytes> from the previous line. It

     is a YAML file with statistical information represented a dictionary.



The stats-tube data is a YAML file representing a single dictionary of strings

to scalars. It contains these keys:



 - "name" is the tube's name.



 - "current-jobs-urgent" is the number of ready jobs with priority < 1024 in

   this tube.



 - "current-jobs-ready" is the number of jobs in the ready queue in this tube.



 - "current-jobs-reserved" is the number of jobs reserved by all clients in

   this tube.



 - "current-jobs-delayed" is the number of delayed jobs in this tube.



 - "current-jobs-buried" is the number of buried jobs in this tube.



 - "total-jobs" is the cumulative count of jobs created in this tube.



 - "current-waiting" is the number of open connections that have issued a

   reserve command while watching this tube but not yet received a response.



The stats command gives statistical information about the system as a whole.

Its form is:



stats\r\n



The server will respond:



OK <bytes>\r\n

<data>\r\n



 - <bytes> is the size of the following data section in bytes.



 - <data> is a sequence of bytes of length <bytes> from the previous line. It

   is a YAML file with statistical information represented a dictionary.



The stats data for the system is a YAML file representing a single dictionary

of strings to scalars. It contains these keys:



 - "current-jobs-urgent" is the number of ready jobs with priority < 1024.



 - "current-jobs-ready" is the number of jobs in the ready queue.



 - "current-jobs-reserved" is the number of jobs reserved by all clients.



 - "current-jobs-delayed" is the number of delayed jobs.



 - "current-jobs-buried" is the number of buried jobs.



 - "cmd-put" is the cumulative number of put commands.



 - "cmd-peek" is the cumulative number of peek commands.



 - "cmd-peek-ready" is the cumulative number of peek-ready commands.



 - "cmd-peek-delayed" is the cumulative number of peek-delayed commands.



 - "cmd-peek-buried" is the cumulative number of peek-buried commands.



 - "cmd-reserve" is the cumulative number of reserve commands.



 - "cmd-use" is the cumulative number of use commands.



 - "cmd-watch" is the cumulative number of watch commands.



 - "cmd-ignore" is the cumulative number of ignore commands.



 - "cmd-delete" is the cumulative number of delete commands.



 - "cmd-release" is the cumulative number of release commands.



 - "cmd-bury" is the cumulative number of bury commands.



 - "cmd-kick" is the cumulative number of kick commands.



 - "cmd-stats" is the cumulative number of stats commands.



 - "cmd-stats-job" is the cumulative number of stats-job commands.



 - "cmd-stats-tube" is the cumulative number of stats-tube commands.



 - "cmd-list-tubes" is the cumulative number of list-tubes commands.



 - "cmd-list-tube-used" is the cumulative number of list-tube-used commands.



 - "cmd-list-tubes-watched" is the cumulative number of list-tubes-watched

   commands.



 - "job-timeouts" is the cumulative count of times a job has timed out.



 - "total-jobs" is the cumulative count of jobs created.



 - "max-job-size" is the maximum number of bytes in a job.



 - "current-tubes" is the number of currently-existing tubes.



 - "current-connections" is the number of currently open connections.



 - "current-producers" is the number of open connections that have each

   issued at least one put command.



 - "current-workers" is the number of open connections that have each issued

   at least one reserve command.



 - "current-waiting" is the number of open connections that have issued a

   reserve command but not yet received a response.



 - "total-connections" is the cumulative count of connections.



 - "pid" is the process id of the server.



 - "version" is the version string of the server.



 - "rusage-utime" is the accumulated user CPU time of this process in seconds

   and microseconds.



 - "rusage-stime" is the accumulated system CPU time of this process in

   seconds and microseconds.



 - "uptime" is the number of seconds since this server started running.



 - "binlog-oldest-index" is the index of the oldest binlog file needed to

   store the current jobs



 - "binlog-current-index" is the index of the current binlog file being

   written to. If binlog is not active this value will be 0



 - "binlog-max-size" is the maximum size in bytes a binlog file is allowed

   to get before a new binlog file is opened



The list-tubes command returns a list of all existing tubes. Its form is:



list-tubes\r\n



The response is:



OK <bytes>\r\n

<data>\r\n



 - <bytes> is the size of the following data section in bytes.



 - <data> is a sequence of bytes of length <bytes> from the previous line. It

   is a YAML file containing all tube names as a list of strings.



The list-tube-used command returns the tube currently being used by the

client. Its form is:



list-tube-used\r\n



The response is:



USING <tube>\r\n



 - <tube> is the name of the tube being used.



The list-tubes-watched command returns a list tubes currently being watched by

the client. Its form is:



list-tubes-watched\r\n



The response is:



OK <bytes>\r\n

<data>\r\n



 - <bytes> is the size of the following data section in bytes.



 - <data> is a sequence of bytes of length <bytes> from the previous line. It

   is a YAML file containing watched tube names as a list of strings.



Using command:
echo -e "stats\r\n" | nc 0.0.0.0 11300
or telnet 0.0.0.0 11300
stats

Beanstalk console

Admin console for Beanstalk queue server, written in PHP




Features

Common list of servers in config for all users
Each user can add its own personal Beanstalkd server
Full list of available tubes
Complete statistics about jobs in tubes
Realtime auto-update with highlighting of changed values
You can view jobs in ready/delayed/buried states in every tube
You can add/kick/delete jobs in every tube
You can select multiple tubes by regExp and clear them
Ability to Pause tubes
Saved jobs (store sample jobs as a template, kick/edit them, very useful for development)
Customizable UI (code highlighter, choose columns, edit auto refresh seconds, pause tube seconds)


https://github.com/ptrofimov/beanstalk_console

Another monitor :http://mnapoli.fr/phpBeanstalkdAdmin/

Another client library php :https://github.com/anorgan/QuTee

Beanstalkd on heroku

https://github.com/kr/bonegrinder

How to delete all (or most) jobs from a beanstalk tube from the shell

Sometimes I need to delete lots of jobs from my beanstalk tube. In this particular case, I had mistakenly added 3x the jobs every day for the last few weeks. These jobs in thesnitch.site tube is how ZippyKid keeps track of what versions of wordpress our customers are using, ensures their configuration is up to date, and other general housekeeping tasks.

Since we were triple queueing the jobs, it resulted in new sites not getting updated in our client inventory system. These weren’t really causing issues, but we definitely noticed something was strange. I stumbled across this graph of the snitch.site tube, and that explained it.

Now, Beanstalk doesn’t have a built-in way to clear out jobs en masse, and I wasn’t particularly interested in using a library to send a few basic commands.

Instead, what I came up with was a fairly concise expect script, which only uses tools installed on the vast majority of Linux systems:

The Code

#!/usr/bin/expect -f
# Filename: beanstalk-purge
set timeout 1

spawn telnet [lindex $argv 0] [lindex $argv 1]
sleep 1
send "use [lindex $argv 2]\n"
expect "USING"

for {set i 1} {$i < [lindex $argv 3]} { incr i 1 } {
    send_user "Proccessing $i\n"
    expect -re {.*} {}
    send "peek-ready\n"
    expect -re {FOUND (\d*) \d*}
    send "delete $expect_out(1,string)\n"
    expect "DELETED"
}



Usage

beanstalk-purge <host> <port> <tube> <count>



Example

beanstalk-purge 127.0.0.1 11300 snitch.site 35000




Delete the first 35,000 jobs out of the snitch.site tube on the beanstalk server located at 127.0.0.1:11300

http://queues.io/

http://dev.af83.com/2013/03/13/why-you-should-consider-beanstalkd.html

Read More