Intorduction to Gearman
- Quick Intro
- Gearman architecture
A Gearman based architecture has three types of elements:
* Job servers
Job servers are instances of the manager program. They behave as proxies by forwarding job requests from a client to a worker that is currently available.
* Clients
The clients talk to a job server in order to request the execution of job functions.
* Workers
Workers are processes that handle requests to jobs by executing functions of code. Each worker makes available a set of functions of code by registering them with a job server.
This architecture can be made fault tolerant by running more than one job server task per machine managing many workers.
Another interesting feature of Gearman, is that workers functions may be written in many different programming languages, such as C, PHP, RUBY and other well known scripting languages. This way you can choose the right programming language for the task.
For instance you can write a program in C or C++ that executes an heavy task and make it register itself as worker daemon. This way you can call C/C++ libraries from PHP without having to turn them into PHP extensions, thus avoiding to deal with thread safety issues.
* Synchronous tasks
There is a kind of tasks that some Web development applications need to execute, which are relatively expensive in terms of RAM and and CPU usage but just take a relatively short period of time to finish.
While the workers handle these heavy job requests synchronously, the clients wait until the jobs are finished without spending any CPU.
* Asynchronous tasks
Gearman also supports dispatching worker jobs asynchronously. This means that the client process does not wait for the worker job to finish.
Gearman keeps an internal job queue and assigns pending jobs to workers as they become free.
This may be useful for long duration tasks.
Example
A worker which reads CSV probably with lakh’s of records to be inserted in a database with proper validations logic.
$worker = new GearmanWorker();
$worker->addServer( "127.0.0.1", 4730 );
$worker->addFunction( "validate_and_insert_csv", "csv_validation_and_insertion" );
while ( $worker->work() )
{
if ( $worker->returnCode() != GEARMAN_SUCCESS )
{
syslog(LOG_ERR, 'return_code: ' . $worker->returnCode());
break;
}
}
function start_contact_validation( $job )
{
$csv_path = $job->workload();
//your logic for validation and insertion goes here...
}
PHP function which calls makes a gearman-client and makes a asynchronous call to a worker registered by $func_name and passes $args to the worker on job-server at localhost and finally returns you the status.
function start_background_job( $func_name, $args, $server_ip='127.0.0.1' )
{
$client= new GearmanClient();
$client->addServer( $server_ip, 4730 );
do
{
$jobid = $client->doBackground( $func_name, $args );
switch ( $client->returnCode() )
{
case GEARMAN_WORK_DATA:
return "Data: $result";
break;
case GEARMAN_WORK_STATUS:
list( $numerator, $denominator )= $client->doStatus();
return "Status: $numerator/$denominator complete";
break;
case GEARMAN_WORK_FAIL:
return "Failed";
exit;
case GEARMAN_SUCCESS:
return "success";
break;
default:
return "RET: " . $client->returnCode();
exit;
}
}
while ( $client->returnCode() != GEARMAN_SUCCESS );
}
Call to validate_and_insert_csv can be give from the above function as given below.
//Start Background Worker for validation and insertion of csv
$status = start_background_job( 'validate_and_insert_csv', $csv_path );
if ( $status == 'success' )
{
echo 'csv imported successfully';
}
else
{
echo 'sorry,there is some problem while importing the csv.';
echo $status;
}
If you want call worker synchronously just make doBackground in start_background_job function as do and this will become a synchronous call which has a value returned from worker.
- Quick Intro
- Gearman is a simple, fast job queuing server.
- Gearman is an anagram for manager. Gearman does not do any work itself, it just distributes jobs to workers.
- Clients, workers and servers can all be on different boxes.
- Jobs can be synchronous or asynchronous.
- Jobs can have priorities.
- It allows your application to perform tasks in parallel, balance the processing load, and even invoke code written in other languages.
- Gearman architecture
A Gearman based architecture has three types of elements:
* Job servers
Job servers are instances of the manager program. They behave as proxies by forwarding job requests from a client to a worker that is currently available.
* Clients
The clients talk to a job server in order to request the execution of job functions.
* Workers
Workers are processes that handle requests to jobs by executing functions of code. Each worker makes available a set of functions of code by registering them with a job server.
This architecture can be made fault tolerant by running more than one job server task per machine managing many workers.
Another interesting feature of Gearman, is that workers functions may be written in many different programming languages, such as C, PHP, RUBY and other well known scripting languages. This way you can choose the right programming language for the task.
For instance you can write a program in C or C++ that executes an heavy task and make it register itself as worker daemon. This way you can call C/C++ libraries from PHP without having to turn them into PHP extensions, thus avoiding to deal with thread safety issues.
* Synchronous tasks
There is a kind of tasks that some Web development applications need to execute, which are relatively expensive in terms of RAM and and CPU usage but just take a relatively short period of time to finish.
While the workers handle these heavy job requests synchronously, the clients wait until the jobs are finished without spending any CPU.
* Asynchronous tasks
Gearman also supports dispatching worker jobs asynchronously. This means that the client process does not wait for the worker job to finish.
Gearman keeps an internal job queue and assigns pending jobs to workers as they become free.
This may be useful for long duration tasks.
Example
A worker which reads CSV probably with lakh’s of records to be inserted in a database with proper validations logic.
$worker = new GearmanWorker();
$worker->addServer( "127.0.0.1", 4730 );
$worker->addFunction( "validate_and_insert_csv", "csv_validation_and_insertion" );
while ( $worker->work() )
{
if ( $worker->returnCode() != GEARMAN_SUCCESS )
{
syslog(LOG_ERR, 'return_code: ' . $worker->returnCode());
break;
}
}
function start_contact_validation( $job )
{
$csv_path = $job->workload();
//your logic for validation and insertion goes here...
}
PHP function which calls makes a gearman-client and makes a asynchronous call to a worker registered by $func_name and passes $args to the worker on job-server at localhost and finally returns you the status.
function start_background_job( $func_name, $args, $server_ip='127.0.0.1' )
{
$client= new GearmanClient();
$client->addServer( $server_ip, 4730 );
do
{
$jobid = $client->doBackground( $func_name, $args );
switch ( $client->returnCode() )
{
case GEARMAN_WORK_DATA:
return "Data: $result";
break;
case GEARMAN_WORK_STATUS:
list( $numerator, $denominator )= $client->doStatus();
return "Status: $numerator/$denominator complete";
break;
case GEARMAN_WORK_FAIL:
return "Failed";
exit;
case GEARMAN_SUCCESS:
return "success";
break;
default:
return "RET: " . $client->returnCode();
exit;
}
}
while ( $client->returnCode() != GEARMAN_SUCCESS );
}
Call to validate_and_insert_csv can be give from the above function as given below.
//Start Background Worker for validation and insertion of csv
$status = start_background_job( 'validate_and_insert_csv', $csv_path );
if ( $status == 'success' )
{
echo 'csv imported successfully';
}
else
{
echo 'sorry,there is some problem while importing the csv.';
echo $status;
}
If you want call worker synchronously just make doBackground in start_background_job function as do and this will become a synchronous call which has a value returned from worker.