Lab 4: Workers

Deadlines

• See labs page

Lab description

The overall goal of this assignment is to reuse the network of chat clients, also used in the CCHAT lab, to transparently implement distributed computation of computationally intensive workloads.

The basic idea is that clients connected to a server act as workers, each computing some small task that is part of a larger job. This kind of distributed computing is often used for computationally intensive tasks, such as mining Bitcoins or computing protein folding.

Programming language

This lab assignment must be developed in Erlang.

How to install Erlang/OTP in your computer.

Specification

The basic goal is to implement a function cchat:send_job/3 with the following arguments:

1. The server name as a string
2. The one-argument function to be applied to each input
3. The list of inputs

The implementation of send_job applies the one-argument function to each of the inputs. It should take care of splitting this job into tasks, distributing them among the clients connected to the chat server, and of collecting and returning the final results.

If you have a server running with clients connected, a job can be sent as follows:

> cchat:send_job("shire", fun(X) -> X * 3 end, [10, 5, 8]).
[30, 15, 24]

Note that the function to be computed is not restricted to numeric arguments: you should be able to pass any function you want! Here, for example, we use a function from the string library:

> cchat:send_job("shire", fun string:to_upper/1, ["erlang", "is", "fun"]).
["ERLANG", "IS", "FUN"]

The idea is that the function to be computed could be computationally intensive, therefore the server should not block while waiting for the answer (but it’s ok for each of the workers to block while computing their tasks).

Assigning tasks

Tasks should be shared among the available clients already connected to the server. Unlike more sophisticated work-stealing approaches, the assignment of tasks does not need to be dynamic. You can use, or adapt, the following function:

assign_tasks([], _) -> [] ;
assign_tasks(Users, Tasks) ->
  [  {lists:nth(((N-1) rem length(Users)) + 1, Users), Task}
  || {N,Task} <- lists:zip(lists:seq(1,length(Tasks)), Tasks) ].

Which works as follows:

> assign_tasks([u1,u2],[t1,t2,t3]).
[{u1,t1},{u2,t2},{u1,t3}]

Order of results

It is important that the order of the results is correct, meaning that each result corresponds to the input in the same position in the list. Precisely, the server is responsible for handling the collection and reordering of results from workers, and for dispatching the results to the client who sent the job. In particular, you should make sure that something like the following cannot happen:

> cchat:send_job("shire", fun(X) -> X * 3 end, [1,5,10]).
[15,3,30]

The test cases described below specifically check that the order of results is correct.

Timeouts

The system should not timeout if a task takes long to compute. Think asynchronously! When using genserver:request, you can extend the default timeout of 3000 ms like so:

genserver:request(Pid, Data, 5000)

Or, alternatively, remove the timeout altogether with:

genserver:request(Pid, Data, infinity)

Error handling

There are a few things which can go wrong:

• The server is not running or unreachable
• There are no clients connected, and thus no workers available
• One or more clients fail to return a response

We recommend that you implement a reasonable behavior in these cases, but for this lab it is not required for you to handle them.

Code skeleton

Use the same skeleton as CCHAT. See the download instructions there. Note that we do not use the client GUI in this lab.

Assignment

• Implement the function cchat:send_job/3 as described above. This also requires to complete the clients client.erl and the server server.erl functionalities accordingly.

• Make sure that the code passes all the test cases.

• Submission: You should submit your code based on the skeleton code and whatever other files are needed for your solution to work. In addition, you should comment your code so that graders can easily review your submission. Please do not submit compressed archives. Just upload the individual source files which you have worked in.

Test cases

The test cases for this lab include all tests for the CCHAT lab, plus some extra ones designed to check the workers’ functionality. Tests are carried out using EUnit. We have created entries in the Makefile to make life easier for you.

Execute the following to run the worker-specific tests:

make -s run_workers_tests

You should see the following kind of output:

# Test: workers
server startup: Ok
spawning 3 clients
sending input: [4,1,7,3,-2,5,2] (7 tasks)
results: [8,2,14,6,-4,10,4] received in 2413ms
results received correctly: Ok
  Test passed.

The total amount of time taken to receive the results is not very important. What is important is that the results arrive in the correct order, and that the server does not block during the computation.