The Wayback Machine - https://web.archive.org/web/20160229142217/http://code.google.com:80/codejam/distributed_guide.html
Sign in
 
 

Guide To Distributed Code Jam

What's the least I need to know to participate?

There's a lot of important information in the official Terms and the FAQ, and if you're competing, you should read them. However, here we provide the main pieces of information on how to actually compete. We assume you know what Google Code Jam is about (since you need to do well in Google Code Jam round 1 to be qualified for Distributed Code Jam), so here's the bare bones for the distributed contest:

  1. In Distributed Code Jam, you submit code (and not outputs), and we compile and run your code.
  2. Your code will run on multiple computers (nodes). Use the message library to communicate between nodes. You can send at most 1000 messages from each node, and the total size of all messages sent from a node cannot exceed 8MB, unless specified otherwise in the problem statement.
  3. You should not read from standard input. The input will be provided through a library function specified for each problem. Each node will get the same input. You need to include/import the library into your solution, using:
    • #include "problem_name.h" in C and C++
    • import problem_name in Python
    • uses problem_name; in Pascal
    • it gets imported automatically in Java
  4. Exactly one node has to print the correct answer to standard output. The other nodes should print nothing.
  5. When you submit a solution to the small input, you will learn whether it was accepted after two minutes. You cannot submit other solutions in the meantime, so take care!
  6. All nodes have to finish (with the exit code 0) within the time limit given in the problem.
As an example, we provide a sample solution for a very simple problem – output the sum of all the numbers given in the input – in C++, C, Python, Pascal, Java.

Announcements after the practice round

After the practice round, we want to turn your attention to a few things that might not be obvious, and a few things we have fixed since then. If you haven't participated in the practice round, you might want to skip this section.

  • Runtime errors:
    • These include exceeding the memory limit, and trying to read outside the bounds of a received message.
  • Rule violation:
    • Sometimes (most notably in Python) memory limit exceeding triggers language constructs the sandboxer interprets as attempts to break out, and judges as a Rule violation.
    • The "clock" call gets reported as "Rule violation" in C++. We intend to fix that for next year.
    • In Java, having multiple top-level classes caused a Rule Violation verdict during the practice contest. This is fixed now.
    • In the practice round, some runtime errors in Java (most notably reading outside the bounds of a received message) were judged as a Rule violation. We have a fix and are testing it; we will announce if we decided to enable it at the beginning of the Sunday contest.
  • Compilation errors:
    • Compiling test runs was broken for the first 6 hours of the practice round. This was fixed and will remain fixed.
    • A number of you asked for treating compilation errors differently (reporting them faster, not giving a time penalty for them). We tried to incorporate some of those proposals, but they interacted in complex ways with the frontend code, and we decided its too risky. We intend to address this next year.
  • Other issues:
    • There was inconsistency in limiting the size of messages sent between the documentation, our judging system, and the local testing tool. During the Sunday round, we will limit the total size of all messages sent from a node to 8MB, with no limit on individual message size. However, messages larger than 10KB can travel much slower due to network fragmentation, so we still recommend using smaller messages.
    • The dashboard was broken for at least three participants. The issue was datastore corruption by a partially completed request. We identified the offending bug and removed it, we are very sorry for the inconvenience this caused during the practice round.
    • There were two issues with the judging system, both of which caused submissions to be stuck "in progress". We identified and fixed both issues for the Sunday contest, we expect judging to happen within the two-minute period for each problem.
    • Some of you had issues running the testing environment on Windows. One option is to use MinGW instead of cygwin. Make sure to add MinGW's environment path before cygwin's. Then, "python dcj.py test --source sandwich.cpp --nodes 10" works (because we can't use shell scripts in MinGW).

A few more details about the contest, please?

The "message" library

Your program runs on multiple computers (nodes). The message library provides the functions needed for the nodes to identify themselves and communicate. You can find the interface for this library here: C++, C, Python, Pascal, Java and you can find usage examples in the "Sum all numbers" example above.

The number of nodes on which your program runs is given in the problem statement. However, you don't need to copy it from there, the NumberOfNodes function will also give you this number. Each node is identified by an "ID", between 0 and NumberOfNodes() - 1, inclusive. The MyNodeId() function will return the ID of a node.

The nodes use messages to communicate. A message is a sequence of values. A node constructs a message using the PutChar, PutInt and PutLL methods (to append a byte, a 32-bit integer and a 64-bit integer, respectively), and then sends it using the Send method. The target of the message can call Receive, and then can retrieve the values using GetChar, GetInt and GetLL. The order in which the values are retrieved has to be the same as the order in which they were inserted, otherwise behaviour is undefined.

The Send method is non-blocking. That is, when you call Send, the sender does not wait for the receiver to receive the message; it just fires the message off and proceeds. The Receive method, on the other hand, is blocking - that is, when a node calls Receive, it will not return until it actually receives a message. In particular, if the sender never sends anything, the receiver will hang forever, and your solution will be judged as timed out.

Internally, the "message" library has buffers, one buffer for each possible source/target. The buffers for messages on which you called Receive, but didn't yet read the contents; and for the messages which you constructed with Put*, but didn't yet Send, count against your solution's memory limit.

Your submission results

When you submit a solution and it gets judged, you can view the results of the submission in the "View my submissions" section (the link is in the tab on the left). All your submissions will be listed there. The status for each submission can be one of the following:

  • In progress — your submission is still being judged. If this persists for significantly over two minutes, notify the organizers through the "Ask a question" form.
  • Correct — your solution was accepted. Congratulations!
  • Wrong answer — your solution finished successfully, but the output was not the correct answer.
  • Compilation error — your solution failed to compile. Make sure you have correctly included the input library of the problem and the message library.
  • Time limit exceeded — the execution of your solution has exceeded the allocated time limit.
  • Output limit exceeded — your solution exceeded the 1MB output limit on one of the nodes.
  • Runtime error — for instance segmentation fault, Python or Java exceptions.
  • Rule violation — your solution tried to call illegal instructions like spawning new treads or creating new files.
  • Messages count limit exceeded — one of the nodes sent too many messages (remember, the default limit is 1000).
  • Messages size limit exceeded — one of the nodes tried to send too much data. The limit is a total of 8MB from each node (that is, the total size of all messages sent cannot exceed 8MB).

Small and large inputs

Similarly to the main track of Code Jam, Distributed Code Jam has small and large inputs, typically one of each for every problem. The small input is intended to have lower limits or be simpler in some other way.

You can submit solutions to the small input for a problem many times. Each solution you submit will get judged in approximately 2 minutes, and you will only be able to submit a new solution once the previous one is judged. Once a submission to the small input is judged correct, you will get points for this input, and you will get penalty time for every previous incorrect submission, as in the main Code Jam track.

You will still be allowed to submit solutions to the small input after one of your submissions is accepted, although the subsequent submissions will not affect your score or penalty time in any way. You can use this option to test the code you intend to submit for the large input. Note that this option is still subject to the two-minute cooldown period.

For the large input, your solution will get judged at the end of the contest. You can resubmit the large input, and only your last submission will be judged by our judging system. Even if you submit a correct solution, but then submit an incorrect one, you will not get points for the large input, so be careful! Similarly, the penalty time (if your submission is correct), will be calculated from the last submission you send.

As in the main track of Code Jam, the scoreboard will show "optimistic" results for all the large submissions.

How fast is the message transfer?

In the first approximation, around 5ms pass between a message being sent and being received, out of which around 3.7ms is spent "in the network" – that is, not blocking either the sender or the receiver. The details depend on the specifics of the traffic pattern (just as the details of the speed of processing depend on details like memory access patterns).

As examples, we provide a few simple benchmarks:

Technical information

The solutions will be judged on a 64-bit linux system. On each node, the program will get limited time and memory (the limits will be specified in the problem statement), as well as have limited network bandwidth (up to 1000 sent messages and 8MB total size) unless specified otherwise in the problem statement).

Solutions in C++ are allowed to use STL. Every node is allowed to write up to 1MB to stderr, this will be ignored. If a solution exceeds 1MB of stderr or stdout output, it will be judged incorrect. If a solution writes to stdout on more than one node, it will be also judged incorrect.

The solutions should not:

  • spawn new threads or processes
  • execute other programs
  • use inlined assembler (in C/C++)
  • use network functions (like socket) – all network communication should go through the message library
  • open files (in particular, using temporary files is not allowed)
  • threaten the system security
  • wait for user interaction, in particular read from stdin

Any of the above can cause the solution to be judged incorrect with a "Rules Violation" verdict, and can result in disqualification if judged as malicious (based on 7.1C of Terms and Conditions).

The size of the submitted code cannot exceed 100KB, and size of the compiled code (for compiled languages) cannot exceed 10MB. Time of compilation cannot exceed 30 seconds.

The solutions will be compiled as follows:

  • gcc -O2 -std=c99 -static -lm, using gcc version 4.7.2 for C
  • g++ -std=gnu++0x -O2 -static -lm, using gcc version 4.7.2 for C++
  • fpc -O2 -XS, using fpc version 2.6.0-9 for Pascal
  • javac, using openjdk 2.2.5 for Java

Solutions will run using Python 2.7.3 for Python.

Local testing tool

You can test your solution locally without submitting the code and receiving penalty time with our local testing tool. Simply download and unpack one of the packages available below. A few warnings:
  • The tool is provided on an as-is basis, and we can't guarantee it will work as intended on a particular platform.
  • Under the hood, the tool spawns a separate process for each node you require. If you have too many nodes, or they use too much CPU/memory, running the tool can freeze or crash your machine.
  • The runtimes with the tool will likely be different than the runtimes on our testing platform. The tool is intended mainly to allow you to estimate the correctness of your solutions, not their performance.

Usage is quite straightforward:

  • alias dcj='{unpack_directory}/dcj.sh'
  • dcj build --source=solution.c
  • dcj run --executable=./solution --nodes=100 --output=all

You can also run dcj test, which bundles the build and run commands together.

The packages are available here:

In order to use the tool you first need to upack the files in a directory of your convenience. You may read the description of how to run the tool by running the following command: python path_of_directory/dcj.py --help.

In case you encounter any problems or we have not published a package built for your operating system we encourage you to read the README file from the package. You can build the tools yourself, as well as modifying the configuration (config.json) file.

To make Python and Java work on Windows you will have to modify paths and commands in build.py. We are working on a version which would work for common Windows configurations.

You should provide the input to your solution, in a similar format to what you can download from the webpage. More precisely:

  • For C and C++:
    • You should have a problem_name.h file available in the same directory as your solution file that contains the definitions of the input methods.
    • Run dcj test --source your_source.c --nodes 10 to test your program.
  • For Java:
    • Your solution file should be named Main.java.
    • Put the problem_name.java file containing the class defining the input methods in the same directory as your solution.
    • Run dcj test --source Main.java --library problem_name.java --nodes 10 to test your program.
  • For Python:
    • The solution file must be executable.
    • On Linux, the first line of the solution must be #!/usr/bin/python. Substitute your path to Python (preferably 2.7) if it differs.
    • You will need to have the problem_name.py file containing the definitions of the input methods in the same directory as your solution.
    • Run the tests using dcj test --source my_source.py --nodes 10.