Gray Soft / Rusting / Taking Rust to Tasktag:graysoftinc.com,2014-03-20:/posts/1272014-09-09T14:46:37ZJames Edward Gray IIThe 1st Comment on "Taking Rust to Task"tag:graysoftinc.com,2014-09-09:/comments/5582014-09-09T14:46:37Z[Concurrency in Rust](https://www.youtube.com/watch?v=oAZ7F7bqT-o) is a handy talk for learning what's available in Rust for multiprocessing.<p><a href="https://www.youtube.com/watch?v=oAZ7F7bqT-o">Concurrency in Rust</a> is a handy talk for learning what's available in Rust for multiprocessing.</p>James Edward Gray IITaking Rust to Tasktag:graysoftinc.com,2014-09-06:/posts/1272014-09-19T14:54:24ZAn example of me exploring how to do message passing style multiprocessing in Rust.<p>Now that I've reached the point where I can get some Rust code running without asking questions in IRC every five minutes, I really wanted to play with some <em>tasks</em>. Tasks are the way Rust handles multiprocessing code. Under the hood they can map one-to-one with operating system threads or you can use a many-to-one mapping that I'm not ready to go into yet.</p>
<p>Probably one of the most exciting aspect of tasks in Rust, in my opinion, is that <a href="http://carol-nichols.com/2014/07/14/ruby-rust-concurrency/">unsafe use of shared memory is rejected outright as a compile error</a>. That lead me to want to figure out how you communicate correctly. (Spoiler: the same was you do in Ruby: <a href="http://graysoftinc.com/rubies-in-the-rough/sleepy-programs">just pass messages</a>.)</p>
<p>Ready to dive in, I grossly simplified a recent challenge from work and coded it up in Rust. You can get the idea with a glance at <code>main()</code>:</p>
<div class="highlight highlight-rust"><pre><span class="k">use</span> <span class="n">std</span><span class="o">::</span><span class="n">collections</span><span class="o">::</span><span class="n">HashMap</span><span class="p">;</span>
<span class="c1">// ...</span>
<span class="k">fn</span> <span class="n">string_vec</span><span class="p">(</span><span class="n">strs</span><span class="o">:</span> <span class="o">&</span><span class="p">[</span><span class="o">&</span><span class="err">'</span><span class="k">static</span> <span class="k">str</span><span class="p">])</span> <span class="o">-></span> <span class="n">Vec</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">v</span> <span class="o">=</span> <span class="n">Vec</span><span class="o">::</span><span class="n">new</span><span class="p">();</span>
<span class="k">for</span> <span class="n">s</span> <span class="n">in</span> <span class="n">strs</span><span class="p">.</span><span class="n">iter</span><span class="p">()</span> <span class="p">{</span>
<span class="n">v</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">s</span><span class="p">.</span><span class="n">to_string</span><span class="p">());</span>
<span class="p">}</span>
<span class="n">v</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">services</span> <span class="o">=</span> <span class="n">HashMap</span><span class="o">::</span><span class="n">new</span><span class="p">();</span>
<span class="n">services</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="s">"S1"</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span> <span class="n">string_vec</span><span class="p">([</span><span class="s">"A"</span><span class="p">,</span> <span class="s">"B"</span><span class="p">]));</span>
<span class="n">services</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="s">"S2"</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span> <span class="n">string_vec</span><span class="p">([</span><span class="s">"A"</span><span class="p">,</span> <span class="s">"C"</span><span class="p">]));</span>
<span class="n">services</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="s">"S3"</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span> <span class="n">string_vec</span><span class="p">([</span><span class="s">"C"</span><span class="p">,</span> <span class="s">"D"</span><span class="p">,</span> <span class="s">"E"</span><span class="p">,</span> <span class="s">"F"</span><span class="p">]));</span>
<span class="n">services</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="s">"S4"</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span> <span class="n">string_vec</span><span class="p">([</span><span class="s">"D"</span><span class="p">,</span> <span class="s">"B"</span><span class="p">]));</span>
<span class="n">services</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="s">"S5"</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span> <span class="n">string_vec</span><span class="p">([</span><span class="s">"A"</span><span class="p">,</span> <span class="s">"Z"</span><span class="p">]));</span>
<span class="k">let</span> <span class="n">work</span> <span class="o">=</span> <span class="n">Work</span><span class="p">(</span><span class="n">Search</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="s">"A"</span><span class="p">.</span><span class="n">to_string</span><span class="p">(),</span> <span class="s">"B"</span><span class="p">.</span><span class="n">to_string</span><span class="p">()));</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">task_manager</span> <span class="o">=</span> <span class="n">TaskManager</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="n">services</span><span class="p">);</span>
<span class="n">task_manager</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">work</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>The <code>HashMap</code> sets up the story. Let's assume we have various travel services: <em>S1</em>, <em>S2</em>, etc. Each of those services can reach various stops. For example, <em>S1</em> can reach <em>A</em> and <em>B</em>. Now the <code>work</code> variable tells you what we want to do, which is to search for ways to get from <em>A</em> to <em>B</em>.</p>
<p>The twist is handled by <code>TaskManager</code>. It's going to run each service in an isolated separate process (a <em>task</em> in Rust speak) that only knows about its own stops. This means finding non-direct paths must involve inter-process communication.</p>
<p>The <code>string_vec()</code> helper just builds a vector of <code>String</code> objects for me. I'm not using <code>String</code> for its mutability, but it's more of an attempt to clarify ownership without scattering lifetime specifications everywhere. (This may not been ideal. Please remember that I'm still a pretty green Rust programmer.)</p>
<p>If the code I write works, it will find two possible paths for our search and indeed it does:</p>
<pre><code>$ ./pathfinder
Path: A--S1-->B
Path: A--S2-->C--S3-->D--S4-->B
</code></pre>
<p>The rest of this blog post will examine just how it accomplishes this.</p>
<p>Let's begin with a pretty trivial data structure:</p>
<div class="highlight highlight-rust"><pre><span class="cp">#[deriving(Clone)]</span>
<span class="k">struct</span> <span class="n">Path</span> <span class="p">{</span>
<span class="n">from</span><span class="o">:</span> <span class="n">String</span><span class="p">,</span>
<span class="n">to</span><span class="o">:</span> <span class="n">String</span><span class="p">,</span>
<span class="n">service</span><span class="o">:</span> <span class="n">String</span>
<span class="p">}</span>
<span class="k">impl</span> <span class="n">Path</span> <span class="p">{</span>
<span class="k">fn</span> <span class="n">new</span><span class="p">(</span><span class="n">from</span><span class="o">:</span> <span class="n">String</span><span class="p">,</span> <span class="n">to</span><span class="o">:</span> <span class="n">String</span><span class="p">,</span> <span class="n">service</span><span class="o">:</span> <span class="n">String</span><span class="p">)</span> <span class="o">-></span> <span class="n">Path</span> <span class="p">{</span>
<span class="n">Path</span><span class="p">{</span><span class="n">from</span><span class="o">:</span> <span class="n">from</span><span class="p">,</span> <span class="n">to</span><span class="o">:</span> <span class="n">to</span><span class="p">,</span> <span class="n">service</span><span class="o">:</span> <span class="n">service</span><span class="p">}</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">to_string</span><span class="p">(</span><span class="o">&</span><span class="n">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">String</span> <span class="p">{</span>
<span class="n">self</span><span class="p">.</span><span class="n">from</span>
<span class="p">.</span><span class="n">clone</span><span class="p">()</span>
<span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">"--"</span><span class="p">)</span>
<span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">service</span><span class="p">.</span><span class="n">as_slice</span><span class="p">())</span>
<span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">"-->"</span><span class="p">)</span>
<span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">to</span><span class="p">.</span><span class="n">as_slice</span><span class="p">())</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>This just defines a <code>Path</code> as two endpoints and the <code>service</code> that connects them. You can see that the <code>to_string()</code> method gives us the simple ASCII arrow output from the solution.</p>
<p><code>Search</code> builds on <code>Path</code>:</p>
<div class="highlight highlight-rust"><pre><span class="cp">#[deriving(Clone)]</span>
<span class="k">struct</span> <span class="n">Search</span> <span class="p">{</span>
<span class="n">from</span><span class="o">:</span> <span class="n">String</span><span class="p">,</span>
<span class="n">to</span><span class="o">:</span> <span class="n">String</span><span class="p">,</span>
<span class="n">paths</span><span class="o">:</span> <span class="n">Vec</span><span class="o"><</span><span class="n">Path</span><span class="o">></span>
<span class="p">}</span>
<span class="k">impl</span> <span class="n">Search</span> <span class="p">{</span>
<span class="k">fn</span> <span class="n">new</span><span class="p">(</span><span class="n">from</span><span class="o">:</span> <span class="n">String</span><span class="p">,</span> <span class="n">to</span><span class="o">:</span> <span class="n">String</span><span class="p">)</span> <span class="o">-></span> <span class="n">Search</span> <span class="p">{</span>
<span class="n">Search</span><span class="p">{</span><span class="n">from</span><span class="o">:</span> <span class="n">from</span><span class="p">,</span> <span class="n">to</span><span class="o">:</span> <span class="n">to</span><span class="p">,</span> <span class="n">paths</span><span class="o">:</span> <span class="n">vec</span><span class="o">!</span><span class="p">[]}</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">services</span><span class="p">(</span><span class="o">&</span><span class="n">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">Vec</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="p">{</span>
<span class="n">self</span><span class="p">.</span><span class="n">paths</span><span class="p">.</span><span class="n">iter</span><span class="p">().</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="n">path</span><span class="o">|</span> <span class="n">path</span><span class="p">.</span><span class="n">service</span><span class="p">.</span><span class="n">clone</span><span class="p">()).</span><span class="n">collect</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">stops</span><span class="p">(</span><span class="o">&</span><span class="n">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">Vec</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">all</span> <span class="o">=</span> <span class="n">vec</span><span class="o">!</span><span class="p">[];</span>
<span class="n">all</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">from</span><span class="p">.</span><span class="n">clone</span><span class="p">());</span>
<span class="n">all</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">to</span><span class="p">.</span><span class="n">clone</span><span class="p">());</span>
<span class="k">for</span> <span class="n">path</span> <span class="n">in</span> <span class="n">self</span><span class="p">.</span><span class="n">paths</span><span class="p">.</span><span class="n">iter</span><span class="p">()</span> <span class="p">{</span>
<span class="n">all</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">path</span><span class="p">.</span><span class="n">from</span><span class="p">.</span><span class="n">clone</span><span class="p">());</span>
<span class="n">all</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">path</span><span class="p">.</span><span class="n">to</span><span class="p">.</span><span class="n">clone</span><span class="p">());</span>
<span class="p">}</span>
<span class="n">all</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">add_path</span><span class="p">(</span><span class="o">&</span><span class="n">self</span><span class="p">,</span> <span class="n">path</span><span class="o">:</span> <span class="n">Path</span><span class="p">)</span> <span class="o">-></span> <span class="n">Search</span> <span class="p">{</span>
<span class="n">Search</span><span class="p">{</span> <span class="n">from</span><span class="o">:</span> <span class="n">path</span><span class="p">.</span><span class="n">to</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span>
<span class="n">to</span><span class="o">:</span> <span class="n">self</span><span class="p">.</span><span class="n">to</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span>
<span class="n">paths</span><span class="o">:</span> <span class="n">self</span><span class="p">.</span><span class="n">paths</span><span class="p">.</span><span class="n">clone</span><span class="p">().</span><span class="n">append</span><span class="p">([</span><span class="n">path</span><span class="p">])</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>A <code>Search</code> holds the endpoints we're currently trying to traverse and any <code>paths</code> collected so far in an attempt to reach our goal. The <code>add_path()</code> method is used with partial matches. It returns a new <code>Search</code> for the remaining distance with the partial <code>Path</code> added to the list.</p>
<p>The other methods, <code>services()</code> and <code>stops()</code>, just return lists of what has been used so far in the accumulated paths. These are helpful in avoiding building circular paths.</p>
<p>Note that both data structures so far are cloneable. This is so that copies can be made to sent across channels to other tasks.</p>
<p>We need two more aggregate data structures to be the messages that we will shuttle between tasks:</p>
<div class="highlight highlight-rust"><pre><span class="cp">#[deriving(Clone)]</span>
<span class="k">enum</span> <span class="n">Job</span> <span class="p">{</span>
<span class="n">Work</span><span class="p">(</span><span class="n">Search</span><span class="p">),</span>
<span class="n">Finish</span>
<span class="p">}</span>
<span class="k">enum</span> <span class="n">Event</span> <span class="p">{</span>
<span class="n">Match</span><span class="p">(</span><span class="n">Vec</span><span class="o"><</span><span class="n">Path</span><span class="o">></span><span class="p">),</span>
<span class="n">Partial</span><span class="p">(</span><span class="n">Vec</span><span class="o"><</span><span class="n">Search</span><span class="o">></span><span class="p">),</span>
<span class="n">Done</span><span class="p">(</span><span class="n">String</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
<p>A <code>Job</code> can be either a wrapped <code>Search</code> we want to perform or the special <code>Finish</code> flag. The latter just tells a task to bust out of its infinite listening-for-searches loop and exit cleanly. This <code>enum</code> lists the messages we can send <strong>to</strong> the service tasks.</p>
<p>Messages <strong>from</strong> a service task (back to our not yet examined <code>TaskManager</code>) are called an <code>Event</code>. There are three possible types. <code>Match</code> means we have a full solution, using the included paths. <code>Partial</code> means that the task matched part of the path and is sending back a list of searches to try for locating the rest of it. Again <code>Done</code> is a special flag that pretty much means, "I've got nothing." This flag includes the name of the service for a reason that will become obvious after we view this next helper object:</p>
<div class="highlight highlight-rust"><pre><span class="k">struct</span> <span class="n">SearchTracker</span> <span class="p">{</span>
<span class="n">counts</span><span class="o">:</span> <span class="n">HashMap</span><span class="o"><</span><span class="n">String</span><span class="p">,</span> <span class="k">uint</span><span class="o">></span>
<span class="p">}</span>
<span class="k">impl</span> <span class="n">SearchTracker</span> <span class="p">{</span>
<span class="k">fn</span> <span class="n">new</span><span class="o"><</span><span class="err">'</span><span class="n">a</span><span class="p">,</span> <span class="n">I</span><span class="o">:</span> <span class="n">Iterator</span><span class="o"><&</span><span class="err">'</span><span class="n">a</span> <span class="n">String</span><span class="o">>></span><span class="p">(</span><span class="k">mut</span> <span class="n">service_names</span><span class="o">:</span> <span class="n">I</span><span class="p">)</span> <span class="o">-></span> <span class="n">SearchTracker</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">tracker</span> <span class="o">=</span> <span class="n">SearchTracker</span><span class="p">{</span><span class="n">counts</span><span class="o">:</span> <span class="n">HashMap</span><span class="o">::</span><span class="n">new</span><span class="p">()};</span>
<span class="k">for</span> <span class="n">name</span> <span class="n">in</span> <span class="n">service_names</span> <span class="p">{</span>
<span class="n">tracker</span><span class="p">.</span><span class="n">counts</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="n">name</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span> <span class="m">0</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">tracker</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">add_search</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">self</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="n">_</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span> <span class="n">in</span> <span class="n">self</span><span class="p">.</span><span class="n">counts</span><span class="p">.</span><span class="n">mut_iter</span><span class="p">()</span> <span class="p">{</span>
<span class="o">*</span><span class="n">count</span> <span class="o">+=</span> <span class="m">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">mark_done</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">self</span><span class="p">,</span> <span class="n">name</span><span class="o">:</span> <span class="n">String</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">count</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="n">counts</span><span class="p">.</span><span class="n">get_mut</span><span class="p">(</span><span class="o">&</span><span class="n">name</span><span class="p">);</span>
<span class="o">*</span><span class="n">count</span> <span class="o">-=</span> <span class="m">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">is_done</span><span class="p">(</span><span class="o">&</span><span class="n">self</span><span class="p">)</span> <span class="o">-></span> <span class="n">bool</span> <span class="p">{</span>
<span class="n">self</span><span class="p">.</span><span class="n">counts</span><span class="p">.</span><span class="n">values</span><span class="p">().</span><span class="n">all</span><span class="p">(</span><span class="o">|</span><span class="n">n</span><span class="o">|</span> <span class="o">*</span><span class="n">n</span> <span class="o">==</span> <span class="m">0</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>The idea behind <code>SearchTracker</code> is that we want to know when we're done seeing results from the various tasks. We can do that by expecting exactly one response (an <code>Event</code>) from each task for each <code>Search</code> sent.</p>
<p>This object just makes a counter for each service name passed into the constructor. You can then <code>add_search()</code> to bump all counters (because each <code>Search</code> is sent to all service tasks) and <code>mark_done()</code> to reduce a named counter when you receive a response from that service. The <code>is_done()</code> method will return true when all the counts balance out.</p>
<p>We need one last helper before we get into the actual process:</p>
<div class="highlight highlight-rust"><pre><span class="k">struct</span> <span class="n">MultiTaskSender</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span>
<span class="n">senders</span><span class="o">:</span> <span class="n">Vec</span><span class="o"><</span><span class="n">Sender</span><span class="o"><</span><span class="n">T</span><span class="o">>></span>
<span class="p">}</span>
<span class="k">impl</span><span class="o"><</span><span class="n">T</span><span class="o">:</span> <span class="n">Clone</span> <span class="o">+</span> <span class="n">Send</span><span class="o">></span> <span class="n">MultiTaskSender</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span>
<span class="k">fn</span> <span class="n">new</span><span class="p">()</span> <span class="o">-></span> <span class="n">MultiTaskSender</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span>
<span class="n">MultiTaskSender</span><span class="p">{</span><span class="n">senders</span><span class="o">:</span> <span class="n">vec</span><span class="o">!</span><span class="p">[]}</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">add_sender</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">self</span><span class="p">,</span> <span class="n">sender</span><span class="o">:</span> <span class="n">Sender</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">)</span> <span class="p">{</span>
<span class="n">self</span><span class="p">.</span><span class="n">senders</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">sender</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">send</span><span class="p">(</span><span class="o">&</span><span class="n">self</span><span class="p">,</span> <span class="n">t</span><span class="o">:</span> <span class="n">T</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="n">sender</span> <span class="n">in</span> <span class="n">self</span><span class="p">.</span><span class="n">senders</span><span class="p">.</span><span class="n">iter</span><span class="p">()</span> <span class="p">{</span>
<span class="n">sender</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">t</span><span class="p">.</span><span class="n">clone</span><span class="p">());</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>As I said, each <code>Search</code> will be sent to all tasks. This simple multicaster just allows you to <code>add_sender()</code> as each task is built and later <code>send()</code> to all those channels. The actual object sent is handled as a generic type that we just need to be able to <code>Clone</code> and <code>Send</code>.</p>
<p>We're finally ready for the main workhorse of this code:</p>
<div class="highlight highlight-rust"><pre><span class="k">struct</span> <span class="n">TaskManager</span> <span class="p">{</span>
<span class="n">services</span><span class="o">:</span> <span class="n">HashMap</span><span class="o"><</span><span class="n">String</span><span class="p">,</span> <span class="n">Vec</span><span class="o"><</span><span class="n">String</span><span class="o">>></span><span class="p">,</span>
<span class="n">tracker</span><span class="o">:</span> <span class="n">SearchTracker</span><span class="p">,</span>
<span class="n">multi_sender</span><span class="o">:</span> <span class="n">MultiTaskSender</span><span class="o"><</span><span class="n">Job</span><span class="o">></span><span class="p">,</span>
<span class="n">event_sender</span><span class="o">:</span> <span class="n">Sender</span><span class="o"><</span><span class="n">Event</span><span class="o">></span><span class="p">,</span>
<span class="n">event_receiver</span><span class="o">:</span> <span class="n">Receiver</span><span class="o"><</span><span class="n">Event</span><span class="o">></span>
<span class="p">}</span>
<span class="k">impl</span> <span class="n">TaskManager</span> <span class="p">{</span>
<span class="k">fn</span> <span class="n">new</span><span class="p">(</span><span class="n">services</span><span class="o">:</span> <span class="n">HashMap</span><span class="o"><</span><span class="n">String</span><span class="p">,</span> <span class="n">Vec</span><span class="o"><</span><span class="n">String</span><span class="o">>></span><span class="p">)</span> <span class="o">-></span> <span class="n">TaskManager</span> <span class="p">{</span>
<span class="k">let</span> <span class="p">(</span><span class="n">sender</span><span class="p">,</span> <span class="n">receiver</span><span class="p">)</span> <span class="o">=</span> <span class="n">channel</span><span class="p">();</span>
<span class="n">TaskManager</span><span class="p">{</span> <span class="n">services</span><span class="o">:</span> <span class="n">services</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span>
<span class="n">tracker</span><span class="o">:</span> <span class="n">SearchTracker</span><span class="o">::</span><span class="n">new</span><span class="p">(</span><span class="n">services</span><span class="p">.</span><span class="n">keys</span><span class="p">()),</span>
<span class="n">multi_sender</span><span class="o">:</span> <span class="n">MultiTaskSender</span><span class="o">::</span><span class="n">new</span><span class="p">(),</span>
<span class="n">event_sender</span><span class="o">:</span> <span class="n">sender</span><span class="p">,</span>
<span class="n">event_receiver</span><span class="o">:</span> <span class="n">receiver</span> <span class="p">}</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="n">run</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">self</span><span class="p">,</span> <span class="n">work</span><span class="o">:</span> <span class="n">Job</span><span class="p">)</span> <span class="p">{</span>
<span class="n">self</span><span class="p">.</span><span class="n">launch_services</span><span class="p">();</span>
<span class="n">self</span><span class="p">.</span><span class="n">send_job</span><span class="p">(</span><span class="n">work</span><span class="p">);</span>
<span class="n">self</span><span class="p">.</span><span class="n">wait_for_services</span><span class="p">();</span>
<span class="n">self</span><span class="p">.</span><span class="n">send_job</span><span class="p">(</span><span class="n">Finish</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</pre></div>
<p>This shows you what a <code>TaskManager</code> keeps track of, which is just the data for our challenge, the helper objects, and various channels (the pipes of communication between Rust tasks). You can see how this gets setup in <code>new()</code>.</p>
<p>Once we have everything we need to track, <code>run()</code> actually does the <code>Job</code>. It will:</p>
<ol>
<li>Launch a task for each service</li>
<li>Send the full search we want to perform</li>
<li>Wait for and respond to reported work from the tasks</li>
<li>Signal all tasks to shutdown when the work is done</li>
</ol><p>Two pieces of this process are beating heart of the system. Here's the first of those:</p>
<div class="highlight highlight-rust"><pre><span class="k">impl</span> <span class="n">TaskManager</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="k">fn</span> <span class="n">launch_services</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">self</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">stops</span><span class="p">)</span> <span class="n">in</span> <span class="n">self</span><span class="p">.</span><span class="n">services</span><span class="p">.</span><span class="n">clone</span><span class="p">().</span><span class="n">move_iter</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">task_event_sender</span> <span class="o">=</span> <span class="n">self</span><span class="p">.</span><span class="n">event_sender</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span>
<span class="k">let</span> <span class="p">(</span><span class="n">search_sender</span><span class="p">,</span> <span class="n">search_receiver</span><span class="p">)</span> <span class="o">=</span> <span class="n">channel</span><span class="p">();</span>
<span class="n">self</span><span class="p">.</span><span class="n">multi_sender</span><span class="p">.</span><span class="n">add_sender</span><span class="p">(</span><span class="n">search_sender</span><span class="p">.</span><span class="n">clone</span><span class="p">());</span>
<span class="n">spawn</span><span class="p">(</span> <span class="n">proc</span><span class="p">()</span> <span class="p">{</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">job</span> <span class="o">=</span> <span class="n">search_receiver</span><span class="p">.</span><span class="n">recv</span><span class="p">();</span>
<span class="k">match</span> <span class="n">job</span> <span class="p">{</span>
<span class="n">Work</span><span class="p">(</span><span class="n">search</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">stops</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="o">&</span><span class="n">search</span><span class="p">.</span><span class="n">from</span><span class="p">)</span> <span class="o">&&</span>
<span class="n">stops</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="o">&</span><span class="n">search</span><span class="p">.</span><span class="n">to</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">path</span> <span class="o">=</span> <span class="n">Path</span><span class="o">::</span><span class="n">new</span><span class="p">(</span>
<span class="n">search</span><span class="p">.</span><span class="n">from</span><span class="p">,</span>
<span class="n">search</span><span class="p">.</span><span class="n">to</span><span class="p">,</span>
<span class="n">name</span><span class="p">.</span><span class="n">clone</span><span class="p">()</span>
<span class="p">);</span>
<span class="k">let</span> <span class="n">paths</span> <span class="o">=</span> <span class="n">search</span><span class="p">.</span><span class="n">paths</span><span class="p">.</span><span class="n">append</span><span class="p">([</span><span class="n">path</span><span class="p">]);</span>
<span class="n">task_event_sender</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">Match</span><span class="p">(</span><span class="n">paths</span><span class="p">))</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">tos</span> <span class="o">=</span> <span class="n">stops</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span>
<span class="k">let</span> <span class="n">previous</span> <span class="o">=</span> <span class="n">search</span><span class="p">.</span><span class="n">stops</span><span class="p">();</span>
<span class="n">tos</span><span class="p">.</span><span class="n">retain</span><span class="p">(</span><span class="o">|</span><span class="n">stop</span><span class="o">|</span> <span class="o">!</span><span class="n">previous</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="n">stop</span><span class="p">));</span>
<span class="k">if</span> <span class="o">!</span><span class="n">search</span><span class="p">.</span><span class="n">services</span><span class="p">().</span><span class="n">contains</span><span class="p">(</span><span class="o">&</span><span class="n">name</span><span class="p">)</span> <span class="o">&&</span>
<span class="n">stops</span><span class="p">.</span><span class="n">contains</span><span class="p">(</span><span class="o">&</span><span class="n">search</span><span class="p">.</span><span class="n">from</span><span class="p">)</span> <span class="o">&&</span>
<span class="o">!</span><span class="n">tos</span><span class="p">.</span><span class="n">is_empty</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">searches</span> <span class="o">=</span> <span class="n">tos</span><span class="p">.</span><span class="n">iter</span><span class="p">().</span><span class="n">map</span><span class="p">(</span> <span class="o">|</span><span class="n">to</span><span class="o">|</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">path</span> <span class="o">=</span> <span class="n">Path</span><span class="o">::</span><span class="n">new</span><span class="p">(</span>
<span class="n">search</span><span class="p">.</span><span class="n">from</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span>
<span class="n">to</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span>
<span class="n">name</span><span class="p">.</span><span class="n">clone</span><span class="p">()</span>
<span class="p">);</span>
<span class="n">search</span><span class="p">.</span><span class="n">add_path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="p">}</span> <span class="p">).</span><span class="n">collect</span><span class="p">();</span>
<span class="n">task_event_sender</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">Partial</span><span class="p">(</span><span class="n">searches</span><span class="p">));</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="n">task_event_sender</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">Done</span><span class="p">(</span><span class="n">name</span><span class="p">.</span><span class="n">clone</span><span class="p">()));</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">Finish</span> <span class="o">=></span> <span class="p">{</span> <span class="k">break</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span> <span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</pre></div>
<p>You're pretty much looking at a service task here (the whole part inside the call to <code>spawn()</code>). Inside, they are just an endless <code>loop</code> calling <code>recv()</code> to get new <code>Work</code> wrapped <code>Search</code> objects from the channel. The first <code>if</code> branch inside the <code>match</code> of <code>Work(search)</code> handles the simple case of the service matching the <code>Search</code> exactly.</p>
<p>When the <code>else</code> branch is selected, because we don't have a direct match, some work is done to see if a partial match is possible. (This is a foolish algorithm, by the way. It does partial matches if it can directly match the <code>from</code> endpoint and not the <code>to</code>. This rules out some viable scenarios, but it helped to keep this already large example smaller.)</p>
<p>If a partial match is found, it's transformed into a list of new searches to try that may later find direct matches or more partial matches. When no direct or partial match is found, the <code>Done</code> flag is sent back so <code>TaskManager</code> knows to stop waiting on this task.</p>
<p>The <code>Finish</code> <code>match</code> clause just breaks out of the <code>loop</code> as described previously. Outside of <code>spawn()</code> is a simple loop that creates each task and some code that prepares the variables for the task to capture.</p>
<div class="highlight highlight-rust"><pre><span class="k">impl</span> <span class="n">TaskManager</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="k">fn</span> <span class="n">wait_for_services</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">self</span><span class="p">)</span> <span class="p">{</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">self</span><span class="p">.</span><span class="n">event_receiver</span><span class="p">.</span><span class="n">recv</span><span class="p">()</span> <span class="p">{</span>
<span class="n">Match</span><span class="p">(</span><span class="n">paths</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">name</span> <span class="o">=</span> <span class="n">paths</span><span class="p">.</span><span class="n">last</span><span class="p">().</span><span class="n">expect</span><span class="p">(</span><span class="s">"No path"</span><span class="p">).</span><span class="n">service</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span>
<span class="n">self</span><span class="p">.</span><span class="n">tracker</span><span class="p">.</span><span class="n">mark_done</span><span class="p">(</span><span class="n">name</span><span class="p">);</span>
<span class="k">let</span> <span class="n">path_string</span> <span class="o">=</span> <span class="n">paths</span><span class="p">.</span><span class="n">iter</span><span class="p">().</span><span class="n">skip</span><span class="p">(</span><span class="m">1</span><span class="p">).</span><span class="n">fold</span><span class="p">(</span>
<span class="n">paths</span><span class="p">[</span><span class="m">0</span><span class="p">].</span><span class="n">to_string</span><span class="p">(),</span>
<span class="o">|</span><span class="n">s</span><span class="p">,</span> <span class="n">p</span><span class="o">|</span> <span class="n">s</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">p</span><span class="p">.</span><span class="n">to_string</span><span class="p">().</span><span class="n">as_slice</span><span class="p">().</span><span class="n">slice_from</span><span class="p">(</span><span class="m">1</span><span class="p">))</span>
<span class="p">);</span>
<span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">"Path: {}"</span><span class="p">,</span> <span class="n">path_string</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">Partial</span><span class="p">(</span><span class="n">searches</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">name</span> <span class="o">=</span> <span class="n">searches</span><span class="p">.</span><span class="n">last</span><span class="p">()</span>
<span class="p">.</span><span class="n">expect</span><span class="p">(</span><span class="s">"No search"</span><span class="p">)</span>
<span class="p">.</span><span class="n">paths</span>
<span class="p">.</span><span class="n">last</span><span class="p">()</span>
<span class="p">.</span><span class="n">expect</span><span class="p">(</span><span class="s">"No path"</span><span class="p">)</span>
<span class="p">.</span><span class="n">service</span>
<span class="p">.</span><span class="n">clone</span><span class="p">();</span>
<span class="n">self</span><span class="p">.</span><span class="n">tracker</span><span class="p">.</span><span class="n">mark_done</span><span class="p">(</span><span class="n">name</span><span class="p">);</span>
<span class="k">for</span> <span class="n">search</span> <span class="n">in</span> <span class="n">searches</span><span class="p">.</span><span class="n">iter</span><span class="p">()</span> <span class="p">{</span>
<span class="n">self</span><span class="p">.</span><span class="n">send_job</span><span class="p">(</span><span class="n">Work</span><span class="p">(</span><span class="n">search</span><span class="p">.</span><span class="n">clone</span><span class="p">()));</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">Done</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span> <span class="n">self</span><span class="p">.</span><span class="n">tracker</span><span class="p">.</span><span class="n">mark_done</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="p">}</span>
<span class="p">}</span>
<span class="k">if</span> <span class="n">self</span><span class="p">.</span><span class="n">tracker</span><span class="p">.</span><span class="n">is_done</span><span class="p">()</span> <span class="p">{</span> <span class="k">break</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</pre></div>
<p>This chunk of code is the other half of the puzzle. It's another infinite <code>loop</code> listening on the <code>Event</code> channel. A full <code>Match</code> is pretty printed and a <code>Partial</code> is sent back out to the services in a wave of new searches. Regardless of the <code>Event</code> type, we record the response for the sending service, though where we find the service name varies by case. This allows us to exit this <code>loop</code> and the program when our <code>tracker</code> says we're done.</p>
<p>There's only one final method on <code>TaskManager</code> and it's what actually sends the messages to the service tasks, tracking each new <code>Search</code> as it goes out:</p>
<div class="highlight highlight-rust"><pre><span class="k">impl</span> <span class="n">TaskManager</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="k">fn</span> <span class="n">send_job</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">self</span><span class="p">,</span> <span class="n">job</span><span class="o">:</span> <span class="n">Job</span><span class="p">)</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">job</span> <span class="p">{</span>
<span class="n">Work</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span> <span class="n">self</span><span class="p">.</span><span class="n">tracker</span><span class="p">.</span><span class="n">add_search</span><span class="p">();</span> <span class="p">}</span>
<span class="n">Finish</span> <span class="o">=></span> <span class="p">{</span> <span class="cm">/* do nothing */</span> <span class="p">}</span>
<span class="p">}</span>
<span class="n">self</span><span class="p">.</span><span class="n">multi_sender</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">job</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>You can find <a href="https://github.com/JEG2/learning_rust/blob/be1cb6cca05dbc92368073fd6c7d703df4b98350/pathfinder/pathfinder.rs">the full code</a> on GitHub.</p>James Edward Gray II