tag:blogger.com,1999:blog-26354452097206891212024-03-13T00:26:36.985-07:00ParFuncRyan Newton's personal blog on Parallelism, Functional Programming, and other things Tech.
<br>
<em>
Note: You can see <a href="http://feeds.feedburner.com/RyanAggregated">all my blog posts aggregated here</a>. Or you can find my <a href="http://cs.indiana.edu/~rrnewton/">home page here</a>.
</em>Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.comBlogger18125tag:blogger.com,1999:blog-2635445209720689121.post-50159427010167306692015-09-24T07:57:00.001-07:002015-12-11T18:01:47.044-08:00IU storms ICFP 2015 -- check out our talks<span style="font-family: inherit;">I feel that all our students and postdocs did a great job at ICFP this year. Now, several weeks have passed since Vancouver, and all the videos are available online for your viewing enjoyment.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">My group works on parallel functional programming, and here's what we presented at the conference this year:</span><br />
<ul>
<li>ICFP: <a href="https://www.youtube.com/watch?v=DLzz1jfj6qE&list=PLnqUlCo055hWNtUo1Haoq347VhCqIjs7u&index=21">my talk on adaptive (functional/concurrent) data</a> (<a href="http://www.cs.indiana.edu/~rrnewton/papers/icfp15_adaptive_data.pdf">paper</a>)</li>
<li>ICFP: <a href="https://www.youtube.com/watch?v=0mYesXvsdcc&list=PLnqUlCo055hWNtUo1Haoq347VhCqIjs7u&index=37">Ed Yang's talk on compact normal forms for serializing functional data</a> (<a href="http://www.cs.indiana.edu/~rrnewton/papers/icfp15_compact_normal_form.pdf">paper</a>)</li>
<li>Haskell: <a href="https://www.youtube.com/watch?v=snXhXA5noVc&list=PLnqUlCo055hV5dPC-4VWeXzhI8ooeTsVy&index=18">Trevor McDonell's talk on type safe LLVM codegen for our Accelerate DSL</a></li>
<li>FHPC: Michael Vollmer <a href="https://www.youtube.com/watch?v=_1_3wi_sLKo&index=3&list=PLnqUlCo055hVkjydE7n3DFYAkeFMdmesC">on our first steps in auto-tuning GPU DSL code</a></li>
<li>FHPC: Eric Holk on <a href="https://www.youtube.com/watch?v=gsWW05KAdys&index=4&list=PLnqUlCo055hVkjydE7n3DFYAkeFMdmesC">our first version of Accelerate that splits programs to run on multiple GPUs</a></li>
<li>HIW: Ömer on <a href="https://www.youtube.com/watch?v=gkx-D-7Y1EU&list=PLnqUlCo055hVfNkQHP7z43r10yNo-mc7B&index=5">rebooting supercompilation in GHC</a></li>
<li>HIW (lightning): Ryan Scott <a href="https://www.youtube.com/watch?v=lTpVN8KQOOg&list=PLnqUlCo055hVfNkQHP7z43r10yNo-mc7B&index=15">on better Foldable deriving in GHC 8.0</a></li>
<li>HIW (lightning): me <a href="https://www.youtube.com/watch?v=TT4poCUSf3A&index=12&list=PLnqUlCo055hVfNkQHP7z43r10yNo-mc7B">on why FP needs to focus on data representation</a> as much / more than code optimizations</li>
</ul>
<br />
And outside of the parallelism topic area IU had some other great talks:<br />
<ul>
<li>ICFP: Cameron Swords <a href="https://www.youtube.com/watch?v=OWs1GUc1RkM&list=PLnqUlCo055hWNtUo1Haoq347VhCqIjs7u&index=40">on a general formulations of contracts</a></li>
<li>ICFP: Spenser Bauman on a <a href="https://www.youtube.com/watch?v=U7FzUitfsvg&list=PLnqUlCo055hWNtUo1Haoq347VhCqIjs7u&index=4">faster Racket with a meta-tracing JIT</a></li>
<li>Scheme: Jason Hemann on microKanren with constraints (not video'd unfortunately)</li>
<li>Our undergrad Zach Sullivan presented a poster</li>
</ul>
Finally, one nice surprise at the conference was running into a student we'd never met before who did a poster on comparing <a href="https://hackage.haskell.org/package/accelerate">Accelerate</a> and <a href="https://github.com/svenssonjoel/Obsidian">Obsidian</a> -- -- two GPU DSLs developed partially and wholly by our postdocs Joel and Trevor. Congrats, guys, on getting users and attention! Even better, the student was one of the winners of the poster competition and thus <a href="https://www.youtube.com/watch?v=8cXZTDLC9FY&list=PLnqUlCo055hWNtUo1Haoq347VhCqIjs7u&index=36">got to present a talk at the main ICFP conference</a>. <br />
<br />Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0tag:blogger.com,1999:blog-2635445209720689121.post-69869004619110881242014-09-16T23:28:00.000-07:002014-09-16T23:42:08.257-07:00Apple Time machine: accomplishment unlocked -- Decreased trust!<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOc8fTAjR6zrhBTPEZ9z0dzgG_Y1vshR2S87ho8kvR64J6xUVJQizleed8Btd6rHVbTVHkzvhPSwka3u91A9Rg2jLyH17uiSn2hcc3dgrKMcHboMeW1i3Eh5CGJRjfbiC5rEv7GfoEaH-6/s1600/hero_timemachine_lg_cracked.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOc8fTAjR6zrhBTPEZ9z0dzgG_Y1vshR2S87ho8kvR64J6xUVJQizleed8Btd6rHVbTVHkzvhPSwka3u91A9Rg2jLyH17uiSn2hcc3dgrKMcHboMeW1i3Eh5CGJRjfbiC5rEv7GfoEaH-6/s1600/hero_timemachine_lg_cracked.jpg" /></a></div>
<br />
The saying is true: you only know backup works when you've restored.<br />
<br />
I got a new laptop and restored from time machine backup on an external drive attached to my airport extreme at home. My time machine backups had been going fine for years. It took a long time, and the network got disconnected so I had to restart it once, but eventually claimed to finish.<br />
<br />
And there's the rub it *claimed* to restore, but it only partially restored. It's one thing to error, but why is it Apple UI's <i>lie</i> to me? (For example, my iPhone's iMessage has shown my message as delivered, whereas my wife's showed no sign of the message.)<br />
<br />
For anyone else who runs into a failed Time Machine restore I'd like to write down what I've found as I manually attempt to synchronize critical folders with the backup image.<br />
<ol>
<li>First, my home directory appears complete. The incomplete restorations all have to do with top-level directories like "/foo" and "/bar" that are either missing entirely or have the following problems.</li>
<li>Some subdirectories are just missing. My Aperture photo library had most years but was totally missing 2012 and 2013, as well as other misc files.</li>
<li>Some files were present but zero length.</li>
<li>Finally, the weird thing is that some files were actually corrupted. I found photos that were smaller size in the restore than in the latest backup image on the time machine disk. Indeed, these jpegs, could still be viewed but would be missing part of the image. </li>
</ol>
The last makes me especially nervous because it's NOT consistent with a (sequential) transfer of files and directories that got interrupted at a single point in time. Of course, I was guilty of restarting the restore process as I'd mentioned at the top. So maybe the bottom line here is that restarting it without manually wiping the HD in between is not safe. (It's not idempotent.)<br />
<br />
Right now I'm using Unison to do the repairs, but for greatest peace of mind I should probably wipe it and restore again... The problem is that I used this machine for a while before noticing that the time machine restore was a partial failure!!Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0tag:blogger.com,1999:blog-2635445209720689121.post-30295980024317152562014-01-06T11:19:00.000-08:002014-01-06T12:46:52.310-08:00Five flavors of the parallel AND operation<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
<style type="text/css">code{white-space: pre;}</style>
<style type="text/css">
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; background-color: #2a211c; color: #bdae9d; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; background-color: #2a211c; color: #bdae9d; border-right: 1px solid #bdae9d; }
td.sourceCode { padding-left: 5px; }
pre, code { color: #bdae9d; background-color: #2a211c; }
code > span.kw { color: #43a8ed; font-weight: bold; }
code > span.dt { text-decoration: underline; }
code > span.dv { color: #44aa43; }
code > span.bn { color: #44aa43; }
code > span.fl { color: #44aa43; }
code > span.ch { color: #049b0a; }
code > span.st { color: #049b0a; }
code > span.co { color: #0066ff; font-style: italic; }
code > span.al { color: #ffff00; }
code > span.fu { color: #ff9358; font-weight: bold; }
code > span.er { font-weight: bold; }
</style>
<link href="hscolour.css" rel="stylesheet" type="text/css"></link>
</head>
<body>
The issue of parallel, logical <code>and</code> operations was recently discussed at length on <a href="http://composition.al/blog/2013/12/24/the-lvar-that-wasnt/">my student, Lindsey Kuper’s, blog</a>.<br />
This is a follow-up to that post, with the purpose of laying out a taxonomy of conjunction operations that might occur in a parallel language. (The same discussion applies equally to disjunction.) Below I classify these operations into five levels, from basic to full featured. A good context to imagine as we go through this is a computationally intensive search problem with lots of nested parallel conjunctions and disjunctions that expose opportunities for parallelism.<br />
Before we get started, some ground rules. We need the operation to take some kind of deferred computations or thunks. Further, we assume these can be turned into <em>futures</em> for exposing parallelism. In our case, we use Haskell and the <code>Par</code> monad so this means that we expect something like this:<br />
<pre class="sourceCode literate haskell"><code class="sourceCode haskell">result <span class="ot"><- span=""> asyncAnd comp1 comp2 </-></span></code></pre>
where <code>comp1</code> and <code>comp2</code> are monadic computations that, once executed, return a <code>Bool</code> value. But the distinctions below could just as well apply to a plain, non-monadic <code>and</code> operation in a lazy, pure language — for example, Haskell’s <code>(&&)</code> operator.<br />
<h3 id="level-one">
Level One</h3>
<ol style="list-style-type: decimal;">
<li><strong>Strict</strong>: Evaluate both <code>comp1</code> and <code>comp2</code> in parallel, and then return the logical and of the resulting booleans.</li>
</ol>
This "level one" <code>and</code> is parallel, but is also less work-efficient than the <em>sequential</em> left-to-right short-circuiting <code>&&</code> operation found in most programming languages. It always evaluates both computations fully. Thus, our next upgrade is short-circuiting.<br />
<h3 id="level-two">
Level Two</h3>
Here it helps to generalize a bit and consider an <code>and</code> that takes a list of an arbitrary number of input computations:<br />
<ol start="2" style="list-style-type: decimal;">
<li><strong>Short-circuiting</strong>: for a list of input computations, evaluate them using a machine-specific degree of parallelism, but also <em>stop</em> launching new computations whenever a <code>False</code> is encountered in an already-completed computation.</li>
</ol>
This is an intermediate category which captures operations which have some degree of parallelism but are <em>asymmetric</em>, much like left-to-right sequential <code>(&&)</code>. That is, some input computations have the ability to short-circuit (deschedule) others, but the relationship is not fair; it’s not all-to-all.<br />
You can build an N-way <code>and</code> meeting this limited criteria using (1) a level-one parallel <code>and</code> together with (2) a short-circuiting sequential <code>and</code>. The two operators could be combined by a heuristic strategy for determining how many parallel tasks are appropriate, and then "bottoming out" to the sequential version. Still, as long as the sequential <code>and</code> is used, inputs are handled asymmetrically; and as long as a level-one parallel <code>and</code> is used we will run unnecessary computations that can’t be descheduled or canceled. Indeed, this is a real tension: work-efficiency vs. parallelism.<br />
<h3 id="level-three">
Level Three</h3>
<ol start="3" style="list-style-type: decimal;">
<li><strong>Short-circuiting, symmetric</strong>: all input computations are potentially evaluated in parallel, and any one of them returning <code>False</code> quickly can potentially cause the result of the entire <code>and</code> operation to become available immediately.</li>
</ol>
Ok, so this helps us get early answers. But it doesn’t impose any requirements as to the work-efficiency of the entire operation. That is, it doesn’t mandate that once an N-way <code>and</code> operation returns <code>False</code> that remaining computations will be cancelled in bounded time. Thus our fourth level concerns cancellation.<br />
<h3 id="level-four">
Level Four</h3>
<ol start="4" style="list-style-type: decimal;">
<li><strong>Symmetric with cancellation</strong>: When an early answer is given, all other threads/tasks are told to stop computing because their answers are no longer needed. There exists a bound on how much computation lingering child-computations of the <code>and</code> may do after this point in time.</li>
</ol>
Anecdotally, I think level-four <code>and</code>s are pretty rare in practical parallel languages and libraries, simply because [transitive] task cancellation is not a common feature of modern parallel runtime systems. I would be happy to be wrong about this! So please share any links to standard library documentation that mention level-four parallel <code>and</code>s out there in the wild.<br />
<h3 id="level-five">
Level Five</h3>
Our final stop is a feature that is not strictly a property of the conjunction operator itself, but rather a complementary feature we find useful in some of our applications. It concerns the ability of canceled computations to do useful work, by storing partial results that are computed before cancellation occurs.<br />
In an unrestricted parallel language, canceled computations may have arbitrary side effects, which, in general, <em>race</em> with the cancellation message itself. In a deterministic language such side effects cannot be allowed; rather, it is usually safe to cancel only "pure" computations. There is one exception, however, a <em>memoized pure function</em> provides a "side effect free" way of storing partial work.<br />
<ol start="5" style="list-style-type: decimal;">
<li><strong>Cancellation + memoization</strong>: Canceled threads never provide a final result, but they can contribute to other computations by adding to their memo-tables.</li>
</ol>
One example of an algorithm which benefits from this combination is the subtype-checking algorithm for equi-recursive types in Chapter 21 of <a href="http://www.cis.upenn.edu/~bcpierce/tapl/">Types and Programming Languages</a>. Verifying a subtype relation between two types requires [parallelizable] conjunctions and disjunctions (e.g. when checking product and sum types), but at the same time memoization can help performance because when typechecking real code, the same subtype checks are made repeatedly.<br />
<a href="http://www.cs.indiana.edu/~lkuper/papers/effectzoo-draft.pdf">Our recent draft paper</a> on "taming the parallel effect zoo", discusses the combination of cancellation and memoization further.<br />
Is there a level six? Is there anything else we might ask of our parallel logical operators?<br />
<h2 id="final-thoughts-parallel-conjunctiondisjunction-in-ghc">
Final thoughts: Parallel conjunction/disjunction in GHC</h2>
GHC with "sparks" (futures) can do level-two <code>and</code>s, for example by sparking the arguments to <code>(&&)</code>. But short-circuiting must follow a predetermined order. Further, cancellation is not possible with sparks, because sparked computations cannot be removed from the spark pool.<br />
If one could use internal GHC functions to determine whether an object is a <a href="http://mainisusuallyafunction.blogspot.com/2011/10/thunks-and-lazy-blackholes-introduction.html">THUNK or CONSTR</a> at runtime, it would then be possible to do more flexible short-circuiting, for example, sparking some elements of a <code>[Bool]</code> while executing others on the current thread, and inbetween executions, polling for completion of the already-sparked futures. Yet this road seems unappealing for a number of reasons.<br />
Alternatively, rather than using the spark mechanism, it is certainly possible to use <code>IO</code> and <code>MVar</code>s to implement parallel-and in Haskell, <a href="http://conal.net/blog/posts/functional-concurrency-with-unambiguous-choice">as Conal Elliot does in this post</a>, using an <code>unamb</code> combinator. But <code>forkIO</code> is vastly higher overhead than sparks, with our own <code>Par</code> monad’s futures falling somewhere inbetween. Moreover, our desire to include transitive cancellation and memoization features motivates us to build these operators on a custom <code>Par</code> monad rather than with <code>IO</code> or sparks directly.<br />
</body>
</html>
Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com1tag:blogger.com,1999:blog-2635445209720689121.post-49199273409372497432013-08-21T07:38:00.000-07:002013-08-21T07:38:10.733-07:00Zero to GHC development in an Ubuntu VM in 19 lines of Bash<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta http-equiv="Content-Style-Type" content="text/css" />
<meta name="generator" content="pandoc" />
<title></title>
<style type="text/css">code{white-space: pre;}</style>
<link rel="stylesheet" href="hscolour.css" type="text/css" />
</head>
<body>
<p>If you're like me, you want to contribute to GHC and even have a few small tasks in mind, but since you're an occasional contributor you don't pull, build, and validate it every day. Thus when you come back to it periodically, getting it validating again can be a chore. For example, most recently it failed on my RHEL 6 work machine and Mac OS 10.8 laptop, but worked on my Ubuntu machine.</p>
<p>Given the complexity and fragility of the process, it would be good for us to have more standardized (hermetic) environments so that people can (1) get started easily and (2) reproduce eachother's build outcomes.</p>
<p>The idea of using a VM image for this purpose <a href="http://projects.haskell.org/pipermail/haskell-platform/2009-July/000572.html">was discussed back in 2009</a>. But it seems not to have happened.
To get things moving in this direction, I came up with a script that bootstraps a working GHC development repository inside a <a href="https://www.virtualbox.org/">VirtualBox</a> VM running Ubuntu 12.04 LTS (either server or desktop worked for me).</p>
<p>The following Bash script is all there is to it:</p>
<script src="https://gist.github.com/rrnewton/6295246.js"></script>
<p>Once I figure out how to shrink down an Ubuntu VM to a reasonable size I'll post a VM for download, or publish an <a href="https://aws.amazon.com/amis">Amazon Machine Image (AMI)</a>. Or, if someone knows <a href="http://docs.opscode.com/chef_overview.html">Chef</a> and would like to help me convert the above to a Chef recipe, that would be better than a bash script.</p>
</body>
</html>
Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0tag:blogger.com,1999:blog-2635445209720689121.post-77239930919761413152013-08-18T22:32:00.001-07:002013-08-18T22:38:23.256-07:00Revisiting Google-Drive a year later: Still not ready for intensive use. Benchmark against AeroFS and BitTorrent Sync.<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="margin-left: 1em; margin-right: 1em;">
</div>
<br />
<br />
Last year, I eagerly anticipated the release Google Drive. I had complained a lot about my experiences <a href="http://parfunk.blogspot.com/2011/02/dropbox-semantics-oh-that-there-were.html">with other synchronization software</a>, and fully expected Google to knock this one out of the park. It's an application that should really emphasize Google's strengths: systems software, storage, scaling distributed systems.<br />
<br />
In Spring 2012, I made the leap and moved all my personal cloud storage over to Google, but I ran into too many technical problems and gave up. (Some of these problems I'll detail below.) Now, a year later, I wanted to check in again and see how things are improving.<br />
<br />
I'm afraid I have to report that there are still these major deal breakers for me, and perhaps for other "power users":<br />
<ul>
<li>Scalability isn't there. For example, if you try the simple benchmark of adding a folder with thousands of very small files, you'll see that maximum throughput is a few files per second.</li>
<li>Getting stuck ("Unable to sync") seems common.</li>
<li>Symlinks are still ignored.</li>
</ul>
It's surprising to me when solutions for syncing do <b>not</b> aggregate meta-data for changed files before communicating over the wire (e.g. like rsync). The Google drive API seems to <a href="https://developers.google.com/drive/v1/reference/">encourage per-file remote operations</a>. I heard there is some support for batching, but I'm not sure if that is specific to some Google APIs or generic across them. It would sure help here.<br />
<br />
Of course, these services all work great for storing small numbers of medium sized files. Maybe there's no desire or need to support scaling and more intensive use? Yet, I think even non-techie users may end up with large numbers of small files even if they don't created them directly (e.g. in my Aperture library). For myself, I ultimately want something closer to a distributed file system. For example, I like to edit files within a git checkout locally on my laptop and have them synced to a server where I run the code. This requires three things:<br />
<ul>
<li>Cross platform -- Linux/Mac in my case.</li>
<li>Low latency -- file edits should appear quickly on the other side.</li>
<li>Equally good treatment of large numbers of small files and small numbers of large files.</li>
</ul>
Alas, in spite of the massive increase in the number of cloud-based directory synchronization options, <b>none</b> seem to meet all three of these criteria. Still. I'll go through a list of disqualifying points at the end of this post.<br />
<br />
The end result is that I still use the same solution I did ten years ago. I run "<span style="font-family: Courier New, Courier, monospace;"><a href="http://www.cis.upenn.edu/~bcpierce/unison/">unison</a> -repeat 2</span>" for linking working copies on different machines. The only thing missing is convenient file-system watching via inotify (i.e. OS-driven notification of changes rather than scanning). This is the killer feature that many of the newer cloud offerings have compared to unison, and it is the key to low-latency, as well as the always-on usage model Dropbox-style systems employ. Unison has rudimentary support for integrating with a file-system watcher and I've <i>sporadically</i> had that functionality working, but it was fragile and hard to set up last time I tried it.<br />
<br />
<br />
<a name='more'></a><br />
<h2>
A simple benchmark</h2>
One thing that was frustrating when Google Drive was released was that there were a plethora of articles and blog posts talking about strategic positioning, competition between the big cloud-storage players in pricing and features, etc; and yet I couldn't find technical reviews talking about correctness features like conflict resolution or objective performance (i.e. benchmarks, not "seems snappy"). As a community, do we not have good sites to review software like this, the same way new graphics cards or digital cameras get benchmarked?<br />
<div>
<br /></div>
<div>
This time around I decided to do a simple benchmark myself. I grabbed a small bioinformatics dataset---3255 small text files, adding up to 15 MB---and dropped it into cloud storage. (Bioinformatics software tends to create large numbers of files, in this case per-gene files for a single species.) How long do different solutions take to upload this directory?</div>
<div>
<br /></div>
<div>
As a baseline it took 1.4 seconds to copy it on my laptop local disk-drive (cold cache). Rsync over the LAN took a similar 1.6 seconds. Rsync from my home to a remote machine---through ~500 miles of internet---about 12.5 seconds. (By contrast, "scp" took 264 seconds, providing another good example of good vs. bad small file handling.)</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://docs.google.com/spreadsheet/oimg?key=0AvzAHqQmHo87dC1lTUQwejQxOFFkNm56RzRWUmlYSnc&oid=1&zx=tdhi39nbiqm" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://docs.google.com/spreadsheet/oimg?key=0AvzAHqQmHo87dC1lTUQwejQxOFFkNm56RzRWUmlYSnc&oid=1&zx=tdhi39nbiqm" /></a></div>
This is a test in typical (uncontrolled) conditions with lots of applications open on my Macbook Pro, transferring to either the cloud or a mediocre Linux work-station over the WAN. The total time to upload this folder varied wildly between the softwares. Most of the data-points above were taken by hand with a stop watch, but Google Drive would run into an "Unable to Sync" error and require restarting, so I couldn't time it in one go. I had to estimate the rate at which it was uploading files and extrapolate, which gives an approximate time of an hour (3600 seconds) to complete the sync. At the other end of the spectrum, unison can startup, scan the directories, and do the transfer in 28.6 seconds. <br />
<br />
Having heard of them on a <a href="http://www.jupiterbroadcasting.com/37056/bittorrent-sync-vs-aerofs-las-s26e10/">recent podcast</a>, I also added the new <a href="http://labs.bittorrent.com/experiments/sync.html">BitTorrent Sync</a> and <a href="https://aerofs.com/">AeroFS</a> into the mix as well. These provide a decentralized, peer-to-peer alternative to Google Drive and Dropbox. BitTorrent Sync took 65 seconds on the benchmark, which was comparatively good. Alas BitTorrent Sync also had a raw latency of 12-18 seconds between changing a single file and that file appearing on another machine (even on the LAN). AeroFS on the other hand, another decentralized system, had the best latency I'd seen for single edits (a couple seconds), but much worse throughput for the 3255 file benchmark. Dropbox achieves pretty good single-edit latency (<a href="http://parfunk.blogspot.com/2011/12/wuala-spideroak-and-dropbox-feature.html">5 seconds in previous tests</a>), but only so-so throughput on the benchmark.<br />
<br />
<h2>
The Achilles' heel of each syncing solution</h2>
<div>
Here's a summary of the problems with each file synchronization solution I've tried.</div>
<div>
<ul>
<li>DropBox -- follows symlinks, which is <a href="https://spideroak.com/faq/questions/73/spideroak_and_backup_of_symbolic_links/">crazy</a>.</li>
<li>BitTorrent Sync -- bad latency</li>
<li>AeroFS -- bad throughput</li>
<li>SpiderOak -- often fails to detect changes on Linux/NFS and allows inconsistent states for unbounded periods of time.</li>
<li>Wuala -- slow, expensive file system scans</li>
<li>Unison -- not always on, difficult to achieve FS monitoring / inotify support</li>
<li>Others -- no Linux client (includes Amazon Cloud Drive, Syncplicity, Box Sync, MS Skydrive, Sugar Sync) </li>
</ul>
</div>
Even though they don't fit my bill, I'd be curious if any of the non-Linux clients manage the low-latency, high-throughput combo. Drop me a line if you're aware of one that does!<br />
<br />
Finally, there are other systems I've heard mentioned but haven't tried yet:<br />
<br />
<ul>
<li><a href="http://www.symform.com/">SymForm</a> and other p2p systems</li>
<li><a href="https://one.ubuntu.com/">Ubuntu One</a> -- just found that this has a cross platform client</li>
<li><a href="https://www.cloudme.com/en">Cloud Me</a> -- also a Linux Client</li>
<li>Likewise <a href="http://www.teamdrive.com/">Team Drive</a></li>
<li>Many <a href="http://en.wikipedia.org/wiki/Comparison_of_file_synchronization_software">more on Wikipedia</a></li>
</ul>
<br />
<h2>
Addendum: Screenshots from first round of Google Drive migration</h2>
<br />
<div>
Here's a chronological account of the problems I had when I first migrated to Google Drive. Some of these may now be out of date.<br />
<br />
(1) Many "Can't sync" errors, including many "unknown issue"s which would then work on fine retry. Unfortunately, on the Mac these [hundreds of] errors would be displayed in a small non-resizable window. <br />
<div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto;">
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; margin: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<br />
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQr5PKY96a1WwmmbA4_ORUtzt3uPiLXY4Iacq7PHOrvHQQTv0wVKCOb6HUhPRyMISM3B_VsceXJzMBKd4itVsbjv1pG_Gkv3CbdzbYB7UwRD0lPeMYLD5OwyewfRhJ_jmKwbjVRjadHXln/s1600/01_Tiny_nonresizable_window.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="color: black;"><img border="0" height="224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQr5PKY96a1WwmmbA4_ORUtzt3uPiLXY4Iacq7PHOrvHQQTv0wVKCOb6HUhPRyMISM3B_VsceXJzMBKd4itVsbjv1pG_Gkv3CbdzbYB7UwRD0lPeMYLD5OwyewfRhJ_jmKwbjVRjadHXln/s320/01_Tiny_nonresizable_window.png" width="320" /></span></a></div>
<br /></div>
<div class="separator" style="clear: both;">
<span style="margin-left: 1em; margin-right: 1em;">(2) When it would get into an "Unable to sync" state the Resume button would be greyed out. I would exit / restart instead.</span></div>
<br />
<br />
<span style="color: black;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJdAdjSjFw19SSmqEjzeqYutC59J-VGK48kkllvFL9YGCZpaQonYghiRdFGfgL4Df07SsC9abgs9sIdLpqRLuN24rcLNBLeDj57Ccacjv-gcJ8ovQPsiEvjyeva5O7HkYZi_xu4WN7qJTB/s1600/03_unable_to_sync_resume_disabled.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJdAdjSjFw19SSmqEjzeqYutC59J-VGK48kkllvFL9YGCZpaQonYghiRdFGfgL4Df07SsC9abgs9sIdLpqRLuN24rcLNBLeDj57Ccacjv-gcJ8ovQPsiEvjyeva5O7HkYZi_xu4WN7qJTB/s320/03_unable_to_sync_resume_disabled.png" width="269" /></a></span><br />
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="text-align: start;">(3) Typical overnight progress on small files would amount to 8000 files over 6 hrs, and would result in >700 mysteriously failed files. (Most would work on retry.)</span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6lpo1WtIAcIJ85TO5BK_81AcDlg4DofI31Iy0XOcUu30orglUwDRIUuiuh3LZcT0TfTUnuPoj05qNpbTlSqnJvlJfRkT6u8fApbQ3z8CLtttfKeHPWXpvxwidXu4kSkqdvd_kpG3VJsfm/s1600/10_typical_overnight_progress_6hrs_8K_files_less_than_1G.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="color: black;"><img border="0" height="133" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6lpo1WtIAcIJ85TO5BK_81AcDlg4DofI31Iy0XOcUu30orglUwDRIUuiuh3LZcT0TfTUnuPoj05qNpbTlSqnJvlJfRkT6u8fApbQ3z8CLtttfKeHPWXpvxwidXu4kSkqdvd_kpG3VJsfm/s320/10_typical_overnight_progress_6hrs_8K_files_less_than_1G.png" width="320" /></span></a></div>
<div>
<br />
(4) A minor GUI issue is that the green check mark would appear next to symlinks, which is a bit misleading, since they are actually ignored by the sync client.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYdiE0pBebBvNGCtM8DXIieR4rvvYHphLDrPFeWZdNQwtlPCCN93IZSMerO_FYTr0AYDD1JtploGrww7uaGuG6zw9i5vJn0cHpyp-soXNKeeFK0LVMUKMKPaJwSbhYwt9HN3diq2b4QcL-/s1600/04_claims_to_have_synced_symlink.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="color: black;"><img border="0" height="69" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYdiE0pBebBvNGCtM8DXIieR4rvvYHphLDrPFeWZdNQwtlPCCN93IZSMerO_FYTr0AYDD1JtploGrww7uaGuG6zw9i5vJn0cHpyp-soXNKeeFK0LVMUKMKPaJwSbhYwt9HN3diq2b4QcL-/s400/04_claims_to_have_synced_symlink.png" width="400" /></span></a></div>
<div>
<br /></div>
<div>
(5) I would continue to see errors regarding syncing a file that was completely GONE from the local file system.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWmDUV6CSrOxXhcysYDU9q8Bb3z-xAPE-XVJjruevTn-Zd8TCJlkUvGZJM7wtcnf7uaOoweAE_Sbq1DxiODXrPbGnoNihi-Kpyf7vaJbK5xCVjuOyWmHa-mQ1E1lXqKBojfp8wWMIA0cny/s1600/05_continues_to_complain_about_file_that_is_GONE.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="color: black;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWmDUV6CSrOxXhcysYDU9q8Bb3z-xAPE-XVJjruevTn-Zd8TCJlkUvGZJM7wtcnf7uaOoweAE_Sbq1DxiODXrPbGnoNihi-Kpyf7vaJbK5xCVjuOyWmHa-mQ1E1lXqKBojfp8wWMIA0cny/s640/05_continues_to_complain_about_file_that_is_GONE.png" width="436" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<div>
(6) Another minor GUI issue was that the Finder icons would still include blue "still syncing" icons even while the status bar reported "Sync complete".</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3yjMDKf4tG7RnBrqUtTioEPB5BbaZyqFhgnHdBp9NzTIk5TRPpy4rx6ssvbKWGrXkNJjJF65wBmwRWmOG-lguvumf5449Wfy1E6bxkt6zzcTIvFpxq-fYi2IU_XbVKNB6GLZG1EHaK3nP/s1600/06_says_sync_is_complete_but_Finder_UI_still_shows_blue_spinnies.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="color: black;"><img border="0" height="190" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3yjMDKf4tG7RnBrqUtTioEPB5BbaZyqFhgnHdBp9NzTIk5TRPpy4rx6ssvbKWGrXkNJjJF65wBmwRWmOG-lguvumf5449Wfy1E6bxkt6zzcTIvFpxq-fYi2IU_XbVKNB6GLZG1EHaK3nP/s400/06_says_sync_is_complete_but_Finder_UI_still_shows_blue_spinnies.png" width="400" /></span></a></div>
<div>
<br /></div>
<div>
(7) The client would often use 100-130% CPU, sometimes even when it is in the "Sync complete" state.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxNTqnca2DOeb-buR_kYo8HVpzWSk2HuoJ5bLSMOkIBpo5PucQzZl9aOGTIBEshPuU0SdLRCdBETfA-Me2q4kKD1vzPa3gz6rkTtuQuJd7a-QYsCqbiGz9lJ7Wok7xmrSu0lKWJyONRbSG/s1600/08_using_100percent_cpu_even_after_sync_complete.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="color: black;"><img border="0" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxNTqnca2DOeb-buR_kYo8HVpzWSk2HuoJ5bLSMOkIBpo5PucQzZl9aOGTIBEshPuU0SdLRCdBETfA-Me2q4kKD1vzPa3gz6rkTtuQuJd7a-QYsCqbiGz9lJ7Wok7xmrSu0lKWJyONRbSG/s400/08_using_100percent_cpu_even_after_sync_complete.png" width="400" /></span></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<div>
(8) I observed a 9GB discrepancy between what the desktop app and website reported my current drive usage to be.</div>
<div>
<br /></div>
<div>
(9) I had spurious file duplication event that created identical copies of 150 files and directories. The files were bitwise identical; there was not an actual conflict.</div>
</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1RTaqP39IDLY_Nf77ynFOPqEfb4uKu2TZ6JXskjMSddvH-Cu9xhZp2ZpnPY0AsQmWqYrKJklOAIZJDkxzUel_O422mkx74hYevoEwtvH45MWkBtDp_w8usSMXUKqye5updpKv711mlr_c/s1600/26_duplicated_all_top_level_dirs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="color: black;"><img border="0" height="136" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1RTaqP39IDLY_Nf77ynFOPqEfb4uKu2TZ6JXskjMSddvH-Cu9xhZp2ZpnPY0AsQmWqYrKJklOAIZJDkxzUel_O422mkx74hYevoEwtvH45MWkBtDp_w8usSMXUKqye5updpKv711mlr_c/s400/26_duplicated_all_top_level_dirs.png" width="400" /></span></a></div>
<br />
(10) Finally, I got a bunch of errors, which would culminate in me needing to disconnect the account. But, most surprisingly, re-associating the account <b><i>required</i></b> deleting everything, and re-downloading the entire ~30GB Google Drive directory.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5klwhZtBVT-WmRHcB4zzYC_4VRJUSUV8ctEcizo5x5oTLG_2eTrRmriaN67UyHJAlu-LHqAkusHh0DDL66OIMolY-6fE5xrjJalMvkbIP_TbtGM4WMLFd67Vt-Ig5bpNKYZgABJBQN2QE/s1600/17_First_app_error_right_after_COMPLETING_big_sync.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5klwhZtBVT-WmRHcB4zzYC_4VRJUSUV8ctEcizo5x5oTLG_2eTrRmriaN67UyHJAlu-LHqAkusHh0DDL66OIMolY-6fE5xrjJalMvkbIP_TbtGM4WMLFd67Vt-Ig5bpNKYZgABJBQN2QE/s400/17_First_app_error_right_after_COMPLETING_big_sync.png" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOiKjxFLCly8MiqF4ZNslqaeBdHdLachjVBwcWCWfFp2bDXvJhaOq9NFYc0_2jXKOnFXGYrrGS3xHTm9NEZZM56DZ8teh8yxu8rNUs-tOeWr42lIlB19LxJNVlj01K1Qx9vd6J8aakF3g3/s1600/20_death_by_recurring_error.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="158" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOiKjxFLCly8MiqF4ZNslqaeBdHdLachjVBwcWCWfFp2bDXvJhaOq9NFYc0_2jXKOnFXGYrrGS3xHTm9NEZZM56DZ8teh8yxu8rNUs-tOeWr42lIlB19LxJNVlj01K1Qx9vd6J8aakF3g3/s400/20_death_by_recurring_error.png" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjl1R8i_bpJK-WkqXoZElLzPmvGpRgQLOxa2uPn1OwR6Hfo9ec-Gq3rQSazsMGf3U8T6oBnjzL6NjteToUE-ItNqke5beKzCeoD_9RRLs8KRzbDuUWuXqkrkEmnUV4h0YBLe3tNqeg9uASA/s1600/24_retrying--coup_de_grace.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="157" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjl1R8i_bpJK-WkqXoZElLzPmvGpRgQLOxa2uPn1OwR6Hfo9ec-Gq3rQSazsMGf3U8T6oBnjzL6NjteToUE-ItNqke5beKzCeoD_9RRLs8KRzbDuUWuXqkrkEmnUV4h0YBLe3tNqeg9uASA/s400/24_retrying--coup_de_grace.png" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifIWcZE4151KEYlGptwG1BAHXMyTaeD1SQPV_tpkmoVd9sEZ8SpOdFe7SPSpHGI4eQcgBk-XNHTCuxvO6_GQab5hOkFU6GtESxGxYIysTLF4t1m_zZuMd3Xi9K5-zCucpd2puaRzlTT0sI/s1600/21_what__REDOWNLOAD_my_files.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifIWcZE4151KEYlGptwG1BAHXMyTaeD1SQPV_tpkmoVd9sEZ8SpOdFe7SPSpHGI4eQcgBk-XNHTCuxvO6_GQab5hOkFU6GtESxGxYIysTLF4t1m_zZuMd3Xi9K5-zCucpd2puaRzlTT0sI/s400/21_what__REDOWNLOAD_my_files.png" width="400" /></a></div>
<br />
<div>
<br /></div>
<div>
<br /></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px;">
<br /></div>
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px;">
<br /></div>
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px;">
</div>
</div>
</div>
Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com11tag:blogger.com,1999:blog-2635445209720689121.post-15417424306253461282013-05-29T15:55:00.000-07:002013-05-29T18:04:28.031-07:00Towards Cloud-based, Crowd-sourced benchmarking? (HSBencher)Producing benchmarking plots using text files and gnuplot has always left me disappointed. We never get things as “push button” as we would like. Further, there’s always too much spit and chewing gum involved, and the marginal cost of asking a new question of the data is too high.<br />
<br />
Thus I’m pleased to announce “HSBencher”, a package I just released <a href="http://hackage.haskell.org/package/hsbencher">on Hackage, here</a> (and <a href="https://github.com/rrnewton/HSBencher/">github here</a>), which tackles a small part of this problem. Namely, it handles running benchmarks while varying parameters and then uploading the resulting benchmark data to <a href="https://www.google.com/fusiontables/">Google Fusion Tables</a>.<br />
<br />
The theory is that it should then be easier to: (1) access the data from any program, anywhere; (2) pass around live graphs that are just a URL; and (3) write scripts to analyze the data for trends. In this post I’ll describe how to use the library, and then I’ll speculate a bit about how it could be used to harvest benchmarking data from more machines via “crowd sourcing”.<br />
<br />
But first, a few notes about where HSBencher fits in vis-a-vis other benchmarking frameworks.<br />
<ul>
<li>HSBencher runs coarse granularity benchmarks, where each benchmark run involves launching a new process. This is appropriate for parallelism benchmarks which need to run for a while, and also for inter-language comparisons.</li>
<li>By contrast, the popular <a href="http://hackage.haskell.org/package/criterion">Criterion</a> package is great for fine-grained Haskell benchmarks.</li>
<li>HSBencher is language-agnostic. What it knows about GHC, cabal, and make is encapsulated in plugins called <a href="http://hackage.haskell.org/packages/archive/hsbencher/1.1/doc/html/HSBencher-Types.html#t:BuildMethod">BuildMethods</a>, with the built-in ones <a href="http://hackage.haskell.org/packages/archive/hsbencher/latest/doc/html/HSBencher-Methods.html">here</a>.</li>
<li>In contrast, there are plenty of language-specific benchmarking tools. For example, if you are writing JVM based code, you might use<a href="http://jmeter.apache.org/">JMeter</a>, and then you could even get some support from <a href="http://jenkins-ci.org/">Jenkins CI</a>. Likewise, for Haskell, <a href="http://hackage.haskell.org/package/fibon">Fibon</a> offers some advantages in that it understands GHC and GHC configuration options.</li>
<li>Many <a href="http://www.cs.cmu.edu/~pbbs/">large benchmark suites</a> just come with simple build systems, such as a series of Makefiles. These can be wrapped to work with HSBencher.</li>
</ul>
<br />
<hr />
<h2 id="hsbencher-usage-model">
</h2>
<h2 id="hsbencher-usage-model">
HSBencher Installation</h2>
First, getting the library should be a one-liner, but requires GHC >= 7.4:<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code> cabal install hsbencher -ffusion</code></pre>
If you have problems with the fusion table flag turned on, try using the latest version of handa-gdata <a href="https://code.google.com/p/hgdata/">from here</a> or <a href="https://github.com/rrnewton/hgdata_mirror">here</a>. Also wipe out your <span style="font-family: Courier New, Courier, monospace;">~/.cabal/googleAuthTokens</span> directory. <br />
<br />
(UPDATE: at the moment HSBencher needs handa-gdata-0.6.2, which is pre-release so use the HEAD version.)<br />
<br />
<br class="Apple-interchange-newline" />
<hr />
<h2 id="hsbencher-usage-model">
</h2>
<h2 id="hsbencher-usage-model">
HSBencher Usage Model</h2>
<br />
HSBencher is configured the same way as <a href="http://xmonad.org/">xmonad</a>, by writing an executable Haskell file and importing the library.<br />
<br />
<pre class="sourceCode literate haskell" style="background-color: #2a211c; color: #bdae9d;"><code class="sourceCode haskell"><span class="kw" style="color: #43a8ed; font-weight: bold;"> import</span> HSBencher
main <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> defaultMainWithBechmarks
[ <span class="dt" style="text-decoration: underline;">Benchmark</span> <span class="st" style="color: #049b0a;">"bench1/bench1.cabal"</span> [<span class="st" style="color: #049b0a;">"1000"</span>] <span class="fu" style="color: #ff9358; font-weight: bold;">$</span>
<span class="dt" style="text-decoration: underline;">Or</span> [ <span class="dt" style="text-decoration: underline;">Set</span> <span class="dt" style="text-decoration: underline;">NoMeaning</span> (<span class="dt" style="text-decoration: underline;">RuntimeParam</span> <span class="st" style="color: #049b0a;">"+RTS -qa -RTS"</span>)
, <span class="dt" style="text-decoration: underline;">Set</span> <span class="dt" style="text-decoration: underline;">NoMeaning</span> (<span class="dt" style="text-decoration: underline;">RuntimeEnv</span> <span class="st" style="color: #049b0a;">"HELLO"</span> <span class="st" style="color: #049b0a;">"yes"</span>) ] ]</code></pre>
The resulting executable or script will also take a number of command line arguments, but the main thing is defining the list of benchmarks, and the configuration space associated with each benchmark.<br />
The configuration spaces are created with a nested series of conjunctions and disjunctions (<code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">And</code> and <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">Or</code> constructors). <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">And</code> is used for setting multiple options simultaneously, and <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">Or</code> creates alternatives which will be exhaustively explored. Creating a series of conjoined <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">Or</code>s of course creates a combinatorial explosion of possible configurations to explore.<br />
<br />
The <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">Set</code> form takes two things. The latter argument contains strings that are opaque to the benchmarking system. These are runtime/compile-time command-line arguments as well as environment variable settings. Together these strings implement the desired behavior (see the <a href="http://www.blogger.com/(http://hackage.haskell.org/packages/archive/hsbencher/1.1/doc/html/HSBencher-Types.html#t:ParamSetting">ParamSetting</a> type). But it’s often useful for the benchmarking framework to know what a setting is actually doing in terms of commonly defined concepts like “setting the number of threads”. Hence the first parameter to <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">Set</code> is a machine-readable encoding of this meaning. For example, a meaning of <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">Threads 32</code> attached to a setting will let the benchmark framework know how to file away the benchamrk result in the Google Fusion Table.<br />
<br />
You can run the above script (without Fusion Table upload) using the following:<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code> runghc minimal.hs</code></pre>
That will leave the output data set in a text file (<code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">results_UNAME.dat</code>) with the full log in <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">bench_UNAME.log</code>. For example:<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code><span style="font-size: x-small;"> # TestName Variant NumThreads MinTime MedianTime MaxTime Productivity1 Productivity2 Productivity3
#
# Tue May 28 23:41:55 EDT 2013
# Darwin RN-rMBP.local 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64
# Ran by: rrnewton
# Determined machine to have 8 hardware threads.
#
# Running each test for 1 trial(s).
# Git_Branch: master
# Git_Hash: f6e0677353680f4c6670316c9c91ce729f19ed0e
# Git_Depth: 244
# Path registry: fromList []
bench1 1000 none 0 0.3 0.3 0.3
bench1 1000 none 0 0.3 0.3 0.3</span></code></pre>
We’ll return to this below and see the fuller schema used by the Fusion table output.<br />
<hr />
<h2 id="setting-up-fusion-tables">
Setting up Fusion Tables</h2>
Because you installed with <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">-ffusion</code> above, you already have the Fusion Tables library. Unfortunately, the <a href="http://hackage.haskell.org/packages/archive/handa-gdata/latest/doc/html/Network-Google-FusionTables.html">Haskell bindings for Google fusion tables</a> are quite limited at the moment, and not that robust. I’m afraid right now [2013.05.29] you need to <a href="http://tables.google.com/">create at least one table manually</a> before following the instructions below.<br />
<br />
Once you’re done with that, go ahead and go to the <a href="https://code.google.com/apis/console/%E2%80%8E">Google API Console</a> and click Create project.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbBempKhWXyNWzuy86gFuJAXt8h7AIt0TWsVrERP8z8zty97Aaf4DpDeRejRTG64zd8p4ONAfWVRH1M4HTf1A45YjGWD6M4fcKISK_hVPyXj-AoU2bhjEYYpLEyVuLnRTwLoxqhWkblI9F/s1600/screenshot1_create_project.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="289" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbBempKhWXyNWzuy86gFuJAXt8h7AIt0TWsVrERP8z8zty97Aaf4DpDeRejRTG64zd8p4ONAfWVRH1M4HTf1A45YjGWD6M4fcKISK_hVPyXj-AoU2bhjEYYpLEyVuLnRTwLoxqhWkblI9F/s320/screenshot1_create_project.png" width="320" /></a></div>
<br />
Next you’ll want to flip the switch to turn on “Fusion Tables API”. It will look like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY3scERl3v1ehzrnecgOk1JasPPIe3oCg0BVMIuT0bMe5sCn3c0NjLzStMdGVHBTi8nn5qFC5z9tv-l-ZQTJov5kVmFRIa5OWa2sGn6P44fedsGxscV9CjSz8QHC-LGwDwFI1lJZWZ5Hee/s1600/screenshot2_turn_on_api.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="73" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY3scERl3v1ehzrnecgOk1JasPPIe3oCg0BVMIuT0bMe5sCn3c0NjLzStMdGVHBTi8nn5qFC5z9tv-l-ZQTJov5kVmFRIa5OWa2sGn6P44fedsGxscV9CjSz8QHC-LGwDwFI1lJZWZ5Hee/s400/screenshot2_turn_on_api.png" width="400" /></a></div>
<br />
<br />
Then “Create an OAuth 2.0 client ID…”:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCyetK2WUJb8puoY72udbzNqAEgtODZl8eoMbMqlO2VD9XdzeTw7hhh3bYUKC9kvl3e_7m0EvNd9V1yGiQ8V4SObvpqM4opfzGP2ooSepY3W3Xfw0a-5G3tuZ_eTuG1CtOgsOxP9ovbj9t/s1600/screenshot3_create_client_ID.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="125" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCyetK2WUJb8puoY72udbzNqAEgtODZl8eoMbMqlO2VD9XdzeTw7hhh3bYUKC9kvl3e_7m0EvNd9V1yGiQ8V4SObvpqM4opfzGP2ooSepY3W3Xfw0a-5G3tuZ_eTuG1CtOgsOxP9ovbj9t/s400/screenshot3_create_client_ID.png" width="400" /></a></div>
<br />
<br />
And select “Installed Application”:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_qLSqT7B38Ps9BhhmvRWPvarwZUpxiqWtXyYuQouauUTDbdcnbSVy_Vbn5Gdc28ghpITysJNuPllq6ALBhU44pKl70wy21In4OQuBeMgVsUQEwUArMyP43taiXh27yCyqpmsdDIF9bi2_/s1600/screenshot4_pick_installed_app.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_qLSqT7B38Ps9BhhmvRWPvarwZUpxiqWtXyYuQouauUTDbdcnbSVy_Vbn5Gdc28ghpITysJNuPllq6ALBhU44pKl70wy21In4OQuBeMgVsUQEwUArMyP43taiXh27yCyqpmsdDIF9bi2_/s400/screenshot4_pick_installed_app.png" width="400" /></a></div>
<br />
Finally you’ll see something like this on the “API Access” tab, which lists the Client ID and “secret”:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipM6AsiEW5oHhduzT_8Ezz6hU6Ev7DXlJe6m7RzOrKui-bsigB6t37POjx0Fd0XhiaGRZjLUDvEgajpMtHDz6cM5X9okRgQz_N3ZiVIuUlOsMTba_ZiPCLAkUubc29RzgMDJtRJAPNS_wL/s1600/screenshot5_example_secret.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="140" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipM6AsiEW5oHhduzT_8Ezz6hU6Ev7DXlJe6m7RzOrKui-bsigB6t37POjx0Fd0XhiaGRZjLUDvEgajpMtHDz6cM5X9okRgQz_N3ZiVIuUlOsMTba_ZiPCLAkUubc29RzgMDJtRJAPNS_wL/s400/screenshot5_example_secret.png" width="400" /></a></div>
<br />
You can provide those to your benchmarker on the command line, but it’s often easier to stick them in environment variables like this:<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code><span style="font-size: x-small;">export HSBENCHER_GOOGLE_CLIENTID=916481248329.apps.googleusercontent.com
export HSBENCHER_GOOGLE_CLIENTSECRET=e2DVfwqa8lYuTu-1oKg42iKN</span></code></pre>
After that, you can run the benchmark script again:<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code>runghc minimal.hs --fusion-upload --name=minimal</code></pre>
When that finishes, if you go to <a href="https://drive.google.com/">drive.google.com</a> (and filter for Tables) you should see a new table called “minimal”. And will see a similar record as above:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyTm4pNX2mzcWiobpY8M7dGm309keNVv4H5kp7-kH_O7L486fjZenfT1-ZQKrtl0s-XxCmjGnUihSHY-rZhF8WJ-eoA6TXxcxeNE3cHQbm9AeTDFHA7xK1rp7q5CH8d4dMKXgRcOY-mPi7/s1600/screenshot6_table_view.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="62" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyTm4pNX2mzcWiobpY8M7dGm309keNVv4H5kp7-kH_O7L486fjZenfT1-ZQKrtl0s-XxCmjGnUihSHY-rZhF8WJ-eoA6TXxcxeNE3cHQbm9AeTDFHA7xK1rp7q5CH8d4dMKXgRcOY-mPi7/s400/screenshot6_table_view.png" width="400" /></a></div>
<br />
<br />
The rest of the columns currently included are:<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code><span style="font-size: x-small;">MINTIME_PRODUCTIVITY
MEDIANTIME_PRODUCTIVITY
MAXTIME_PRODUCTIVITY
ALLTIMES
TRIALS
COMPILER
COMPILE_FLAGS
RUNTIME_FLAGS
ENV_VARS
BENCH_VERSION
BENCH_FILE
UNAME
PROCESSOR
TOPOLOGY
GIT_BRANCH
GIT_HASH
GIT_DEPTH
WHO
ETC_ISSUE
LSPCI
FULL_LOG</span></code></pre>
If you play around with Fusion Tables, you’ll find that you can do basic plotting straight off this data, for example, threads vs. time:<br />
<br />
<center>
<iframe frameborder="no" height="300" scrolling="no" src="https://www.google.com/fusiontables/embedviz?containerId=gviz_canvas&viz=GVIZ&t=LINE_AGGREGATE&isXyPlot=true&bsize=0&q=select+col4%2C+col6%2C+col8%2C+col7+from+1w1erplF51yReOhtLVEIK2HTiTc4nd7BDjVLIiO0+where+col0+%3D+'MatMult'+and+col1+%3D+'Direct'&qrs=+and+col4+%3E%3D+&qre=+and+col4+%3C%3D+&qe=+order+by+col4+asc&uiversion=2&gco_forceIFrame=true&gco_hasLabelsColumn=true&width=500&height=300" width="500"></iframe></center>
(Beyond 12 is using hyperthreading on this machine.) Filtering support is great, group-by or other aggregation seems absent currently. Thus you’ll ultimately want to have other scripts that ingest the data, analyze it, and republish it in a summarized form.<br />
<hr />
<h2 id="benchmark-format-and-buildmethods">
Benchmark format and <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">BuildMethods</code></h2>
The Benchmark data structure in the example above specifies a <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">.cabal</code> file as the <i>target</i>, and <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">1000</code> as the command line arguments to the resulting executable. (Note that the <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">RunTimeParams</code> also ultimately get passed as command line arguments to the executable.)<br />
<br />
The convention in HSBencher is that a path uniquely identifies an individual benchmark; properties of that file (or the surrounding files) determine which build methods applies. For <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">.cabal</code> or <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">.hs</code> files, the extensions activates the appropriate build method.<br />
<br />
The interface for build methods is defined <a href="https://github.com/rrnewton/HSBencher/blob/6a8bb919e4ac0a1e8dc221ddeb402c0b65f4bbe9/HSBencher/Methods.hs#L70">here</a>. and the <a href="https://github.com/rrnewton/HSBencher/blob/6a8bb919e4ac0a1e8dc221ddeb402c0b65f4bbe9/HSBencher/Types.hs#L116">full code of the <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">ghc</code> build method</a> is show below:<br />
<pre class="sourceCode literate haskell" style="background-color: #2a211c; color: #bdae9d;"><code class="sourceCode haskell"><span style="font-size: x-small;">ghcMethod <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> <span class="dt" style="text-decoration: underline;">BuildMethod</span>
{ methodName <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> <span class="st" style="color: #049b0a;">"ghc"</span>
, canBuild <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> <span class="dt" style="text-decoration: underline;">WithExtension</span> <span class="st" style="color: #049b0a;">".hs"</span>
, concurrentBuild <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> <span class="kw" style="color: #43a8ed; font-weight: bold;">True</span>
, setThreads <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> <span class="kw" style="color: #43a8ed; font-weight: bold;">Just</span> <span class="fu" style="color: #ff9358; font-weight: bold;">$</span> \ n <span class="ot">-></span> [ <span class="dt" style="text-decoration: underline;">CompileParam</span> <span class="st" style="color: #049b0a;">"-threaded -rtsopts"</span>
, <span class="dt" style="text-decoration: underline;">RuntimeParam</span> (<span class="st" style="color: #049b0a;">"+RTS -N"</span><span class="fu" style="color: #ff9358; font-weight: bold;">++</span> <span class="fu" style="color: #ff9358; font-weight: bold;">show</span> n<span class="fu" style="color: #ff9358; font-weight: bold;">++</span><span class="st" style="color: #049b0a;">" -RTS"</span>)]
, clean <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> \ pathMap bldid target <span class="ot">-></span> <span class="kw" style="color: #43a8ed; font-weight: bold;">do</span>
<span class="kw" style="color: #43a8ed; font-weight: bold;">let</span> buildD <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> <span class="st" style="color: #049b0a;">"buildoutput_"</span> <span class="fu" style="color: #ff9358; font-weight: bold;">++</span> bldid
liftIO<span class="fu" style="color: #ff9358; font-weight: bold;">$</span> <span class="kw" style="color: #43a8ed; font-weight: bold;">do</span> b <span class="ot"><- span=""> doesDirectoryExist buildD
when b<span class="fu" style="color: #ff9358; font-weight: bold;">$</span> removeDirectoryRecursive buildD
<span class="fu" style="color: #ff9358; font-weight: bold;">return</span> ()
, compile <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> \ pathMap bldid flags target <span class="ot">-></span> <span class="kw" style="color: #43a8ed; font-weight: bold;">do</span>
<span class="kw" style="color: #43a8ed; font-weight: bold;">let</span> dir <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> takeDirectory target
file <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> takeBaseName target
suffix <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> <span class="st" style="color: #049b0a;">"_"</span><span class="fu" style="color: #ff9358; font-weight: bold;">++</span>bldid
ghcPath <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> M.findWithDefault <span class="st" style="color: #049b0a;">"ghc"</span> <span class="st" style="color: #049b0a;">"ghc"</span> pathMap
log<span class="fu" style="color: #ff9358; font-weight: bold;">$</span> tag<span class="fu" style="color: #ff9358; font-weight: bold;">++</span><span class="st" style="color: #049b0a;">" Building target with GHC method: "</span><span class="fu" style="color: #ff9358; font-weight: bold;">++show</span> target
inDirectory dir <span class="fu" style="color: #ff9358; font-weight: bold;">$</span> <span class="kw" style="color: #43a8ed; font-weight: bold;">do</span>
<span class="kw" style="color: #43a8ed; font-weight: bold;">let</span> buildD <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> <span class="st" style="color: #049b0a;">"buildoutput_"</span> <span class="fu" style="color: #ff9358; font-weight: bold;">++</span> bldid
liftIO<span class="fu" style="color: #ff9358; font-weight: bold;">$</span> createDirectoryIfMissing <span class="kw" style="color: #43a8ed; font-weight: bold;">True</span> buildD
<span class="kw" style="color: #43a8ed; font-weight: bold;">let</span> dest <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> buildD <span class="fu" style="color: #ff9358; font-weight: bold;"></></span> file <span class="fu" style="color: #ff9358; font-weight: bold;">++</span> suffix
runSuccessful <span class="st" style="color: #049b0a;">" [ghc] "</span> <span class="fu" style="color: #ff9358; font-weight: bold;">$</span>
printf <span class="st" style="color: #049b0a;">"%s %s -outputdir ./%s -o %s %s"</span>
ghcPath file buildD dest (<span class="fu" style="color: #ff9358; font-weight: bold;">unwords</span> flags)
<span class="fu" style="color: #ff9358; font-weight: bold;">return</span> (<span class="dt" style="text-decoration: underline;">StandAloneBinary</span><span class="fu" style="color: #ff9358; font-weight: bold;">$</span> dir <span class="fu" style="color: #ff9358; font-weight: bold;"></></span> dest)
}
<span class="kw" style="color: #43a8ed; font-weight: bold;">where</span>
tag <span class="fu" style="color: #ff9358; font-weight: bold;">=</span> <span class="st" style="color: #049b0a;">" [ghcMethod] "</span><!-----><!-----><!-----></-></span></span></code></pre>
That’s it!<br />
<br />
In this case we use a custom build directory triggered off the build ID so that concurrent builds of different configurations are possible. More details on the conventions that benchmarks should follow can be found in the <a href="https://github.com/rrnewton/HSBencher/blob/master/README.md">README</a>.<br />
<hr />
<h2 id="crowd-sourcing">
Crowd sourcing?</h2>
Performance regression testing, say, for a compiler like GHC, is no easy thing. The surface area is large (libraries, language features), as is the variety of hardware architectures it will run on in the wild. One idea is that if we could make it absolutely trivial for users to run the benchmark suite, then we could aggregate that data in the cloud, and this could provide a variety of combinations of OS, hardware, and libraries.<br />
<br />
Unfortunately, the Google Drive permissions model doesn’t support this well right now. Each Fusion Table is either readable or editable by a particular usable. There’s no way to switch one to “append-only” or even make it world-writable, even if one wanted to live dangerously.<br />
<br />
Nevertheless, it would be possible to follow a “pull request” kind of model where a particular HSBencher-based client uploads to a personal Fusion table, and then sends a request to a daemon that ingests the data into the global, Aggregated fusion table for that project. We’ll see.<br />
<hr />
<h2 id="future-work-and-feedback">
Future work and feedback</h2>
For our own use, I’ve converted the <a href="http://hackage.haskell.org/package/monad-par">monad-par</a> benchmarks, turning them into <a href="https://github.com/simonmar/monad-par/blob/895180548d33ab85f75635231b20ad3c54748d3d/examples/benchmark.hs">this script</a>. I have my eye on the GHC nofib tests as another target. If you get a chance to try HSBencher, let me know how it works for you. It will need many more features, I’m sure, and I’d like to hear which your benchmark suite needs.<br />
<br />Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com1tag:blogger.com,1999:blog-2635445209720689121.post-17898620194857969092013-04-05T23:52:00.002-07:002013-04-05T23:52:37.079-07:00Don't mux parallel shell output! Introducing Hydra-print.<div class="" style="clear: both; text-align: center;">
</div>
<div style="text-align: start;">
</div>
In this age of multicore-everything (your laptop, your phone, your toaster) there’s no excuse for anything not to run in parallel, right? So what about text-based shell programs? On the one hand, it has always been easy to launch things in parallel, say with bash’s support for backgrounding jobs, or using <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">make -j</code>.<br />
<br />
<br />
But, on the other hand, I’ve never been happy with the output of <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">make -j</code>. Interleaving lines of output from parallel subprocesses isn’t very user friendly. (Indeed, <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">cabal -j</code> used to do the same thing, and recently it has switched to simply <i>hiding</i> the output instead.) Thus I’ve written a little utility than dynamically splits the terminal window when the compute job enters a parallel region (video below).<br />
<hr />
<div class="separator" style="clear: both; text-align: center;">
<object class="BLOGGER-youtube-video" classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0" data-thumbnail-src="http://i.ytimg.com/vi/q9my1_hlWW0/0.jpg" height="266" width="320"><param name="movie" value="http://www.youtube.com/v/q9my1_hlWW0?version=3&f=user_uploads&c=google-webdrive-0&app=youtube_gdata" /><param name="bgcolor" value="#FFFFFF" /><param name="allowFullScreen" value="true" /><embed width="320" height="266" src="http://www.youtube.com/v/q9my1_hlWW0?version=3&f=user_uploads&c=google-webdrive-0&app=youtube_gdata" type="application/x-shockwave-flash" allowfullscreen="true"></embed></object></div>
It’s called “hydra-print” and is both a Haskell library and a pair of command-line programs (<code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">hydra-view</code> and <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">hydra-head</code>). You can find it at its <a href="https://github.com/rrnewton/hydra-print">Github page here</a> and the released package is <a href="http://hackage.haskell.org/package/hydra-print">on Hackage here</a>.<br />
<br />
After running <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">cabal install hydra-print</code>, the simplest way to use it is to bring up a server like this:<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code>hydra-view</code></pre>
And then arrange for a script to pipe the output of multiple commands to it in parallel:<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code>(command1 | hydra-head) &
(command2 | hydra-head) &</code></pre>
It’s possible to compose command’s output sequentially as well, by asking <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">hydra-head</code> to return a named pipe, and deleting it to signal that the stream is finished.<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code>p=`hydra-head -p`
command1 >> $p
command2 >> $p
rm -f $p</code></pre>
Further, its often convenient to use hydra-view to launch another command, using it to seed the view with output:<br />
<pre style="background-color: #2a211c; color: #bdae9d;"><code>hydra-view -- make -j</code></pre>
The repository contains an <a href="https://github.com/rrnewton/hydra-print/blob/53f567ef2e4e96a26547fd38cf5a660aa5d25883/examples/make/Makefile">example Makefile here</a>, which shows how to adapt a Makefile to use hydra-print without any direct support builtin to Make. Of course, built-in support is better, and I hope to come up with a patch for <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">cabal-install</code> to use hydra-print soon.<br />
<br />
<h1 id="using-the-haskell-library">
Using the Haskell Library</h1>
The library is based on the recent <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">io-streams</code> package, and makes it very easy to send several kinds of Haskell data streams into hydra-print.<br />
<br />
<pre class="sourceCode literate haskell" style="background-color: #2a211c; color: #bdae9d;"><code class="sourceCode haskell"><span class="kw" style="color: #43a8ed; font-weight: bold;">import</span> <span class="kw" style="color: #43a8ed; font-weight: bold;">qualified</span> System.IO.Streams <span class="kw" style="color: #43a8ed; font-weight: bold;">as</span> S
<span class="kw" style="color: #43a8ed; font-weight: bold;">import</span> UI.HydraPrint</code></pre>
The two main ways to use the library are to (1) bring up a static interface with a given number of windows, or (2) bring up a dynamic view monitoring a changing collectionn of streams.<br />
<br />
The former can be called with <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">hydraPrintStatic defaultHydraConf (zip names strms)</code>. It’s type signature is:<br />
<pre class="sourceCode literate haskell" style="background-color: #2a211c; color: #bdae9d;"><code class="sourceCode haskell"><span class="ot">hydraPrintStatic ::</span> </code><span class="dt" style="text-decoration: underline;">HydraConf</span> <span class="ot">-></span> [(<span class="dt" style="text-decoration: underline;">String</span>, <span class="dt" style="text-decoration: underline;">InputStream</span> <span class="dt" style="text-decoration: underline;">ByteString</span>)] <span class="ot">-></span> <span class="dt" style="text-decoration: underline;">IO</span> ()</pre>
<br />
As you can see, it takes a fixed number of streams (in a list). A dynamic set of streams is represented as – what else – a stream of streams!<br />
<pre class="sourceCode literate haskell" style="background-color: #2a211c; color: #bdae9d;"><code class="sourceCode haskell"><span class="ot">hydraPrint ::</span> </code><span class="dt" style="text-decoration: underline;">HydraConf</span> <span class="ot">-></span> <span class="dt" style="text-decoration: underline;">InputStream</span> (<span class="dt" style="text-decoration: underline;">String</span>, <span class="dt" style="text-decoration: underline;">InputStream</span> <span class="dt" style="text-decoration: underline;">ByteString</span>) <span class="ot">-></span> <span class="dt" style="text-decoration: underline;">IO</span> ()</pre>
<br />
In a parallel program one might use a <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">Control.Concurrent.Chan</code> to aggregate streams and send them to <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">hydraPrint</code>:<br />
<br />
<pre class="sourceCode literate haskell" style="background-color: #2a211c; color: #bdae9d;"><code class="sourceCode haskell">strmSrc <span class="ot"><- span=""> chanToInput strmChan
hydraPrint defaultHydraConf strmSrc<!-----></-></span></code></pre>
<br />
Finally, I think the most obvious improvement to make to this little utility would be to provide the option of a <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">tmux</code> interface instead. This would provide much more functionality and should be pretty straightforward, because <code style="background-color: #2a211c; color: #bdae9d; white-space: pre;">tmux</code> itself is quite scriptable. Emacs [client] would be another option.<br />
<br />
Enjoy!<br />
<br />
<br />
<br />
<br />
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
</head></html>Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0Bloomington, IN, USA39.165325 -86.52638569999999239.066843500000004 -86.68774719999999 39.2638065 -86.3650242tag:blogger.com,1999:blog-2635445209720689121.post-8042142646748847462012-05-04T13:11:00.000-07:002012-05-15T21:30:38.178-07:00How to write hybrid CPU/GPU programs with Haskell<title></title>
<style type="text/css">
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; background-color: #2a211c; color: #bdae9d; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; background-color: #2a211c; color: #bdae9d; border-right: 1px solid #bdae9d; }
td.sourceCode { padding-left: 5px; }
pre, code { color: #bdae9d; background-color: #2a211c; }
code > span.kw { color: #43a8ed; font-weight: bold; }
code > span.dt { text-decoration: underline; }
code > span.dv { color: #44aa43; }
code > span.bn { color: #44aa43; }
code > span.fl { color: #44aa43; }
code > span.ch { color: #049b0a; }
code > span.st { color: #049b0a; }
code > span.co { color: #0066ff; font-style: italic; }
code > span.al { color: #ffff00; }
code > span.fu { color: #ff9358; font-weight: bold; }
code > span.er { font-weight: bold; }
</style>
<br />
What’s better than programming a GPU with a high-level, Haskell-embedded DSL (domain-specific-language)? Well, perhaps writing portable CPU/GPU programs that utilize both pieces of silicon—with dynamic load-balancing between them—would fit the bill.<br />
<br />
This is one of the heterogeneous programming scenarios supported by our <a href="https://github.com/simonmar/monad-par/">new <strong>meta-par</strong> packages</a>. A draft paper <a href="http://www.cs.indiana.edu/~rrnewton/papers/meta-par_submission.pdf">can be found here</a>, which explains the mechanism for building parallel schedulers out of "mix-in" components. In this post, however, we will skip over that and take a look at CPU/GPU programming specifically.<br />
<br />
This post assumes familiarity with the <strong>monad-par</strong> parallel programming library, <a href="http://www.cs.indiana.edu/~rrnewton/papers/haskell2011_monad-par.pdf">which is described in this paper</a>.<br />
<h2 id="getting-started">
Getting Started</h2>
First, we install the just-released <a href="http://hackage.haskell.org/package/meta-par-accelerate"><strong>meta-par-accelerate</strong> package</a>:<br />
<pre><span style="background-color: transparent;">cabal install c2hs
</span><span style="background-color: transparent;">cabal install meta-par-accelerate</span></pre>
If you have CUDA 4.1 installed, then just remove the "-f-cuda" (or remove the entire middle line). And then we import the following module:<br />
<pre class="sourceCode literate haskell"><code class="sourceCode haskell"><span class="kw">import</span> <span class="dt">Control.Monad.Par.Meta.AccSMP</span></code></pre>
This provides a scheduler for (Accelerate GPU-EDSL) + (monad-par multicore CPU) scheduling: It also reexports the <a href="http://www.cs.indiana.edu/~rrnewton/haddock/abstract-par-accelerate/Control-Monad-Par-Accelerate.html#t:ParAccelerate"><strong>ParAccelerate</strong> type class</a> which provides the ability to launch GPU computations from within a <strong>Par</strong> computation.<br />
<br />
Next, we also import Accelerate itself to so that we can express <strong>Acc</strong> computations that can run on the GPU:<br />
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">import</span> <span class="dt">Data.Array.Accelerate</span>
<span class="kw">import</span> <span class="dt">Data.Array.Accelerate.CUDA</span> <span class="kw">as</span> <span class="dt">CUDA</span></code></pre>
(By the way, this blog post is an executable literate Haskell file <a href="https://gist.github.com/2597128">that can be found here</a>.)<br />
Now we are ready to create a trivial Accelerate computation:<br />
<pre class="sourceCode literate haskell"><code class="sourceCode haskell"><span class="ot">triv ::</span> <span class="dt">Acc</span> (<span class="dt">Scalar</span> <span class="dt">Float</span>)
triv <span class="fu">=</span> <span class="kw">let</span> arr <span class="fu">=</span> generate (constant (<span class="dt">Z</span> <span class="fu">:.</span> (<span class="dv">10</span><span class="ot">::</span><span class="dt">Int</span>)))
(\ i <span class="ot">-></span> <span class="fl">3.3</span> )
<span class="kw">in</span> fold (<span class="fu">+</span>) <span class="dv">0</span> arr</code></pre>
This creates a ten element one-dimensional array, and then sums up the elements. We could run this directly using CUDA, which would print out <strong>Array (Z) [33.0]</strong>. That’s just Accelerate’s fancy way of saying "<strong>33.0</strong>"—a scalar is a zero-dimensional array. Here’s how we invoke Accelerate/CUDA:<br />
<pre class="sourceCode literate haskell"><code class="sourceCode haskell">runDirect <span class="fu">=</span> <span class="fu">print</span> (CUDA.run triv)</code></pre>
If we are inside a Par computation, on the other hand, we simply use <strong>runACC</strong> or <strong>runAccWith</strong>:<br />
<pre class="sourceCode literate haskell"><code class="sourceCode haskell">runWPar1 <span class="fu">=</span> <span class="fu">print</span> (runPar (runAcc triv))
runWPar2 <span class="fu">=</span> <span class="fu">print</span> (runPar (runAccWith CUDA.run triv))</code></pre>
The former uses the default Accelerate implementation. The latter specifies which Accelerate implementation to use. After all, there should ultimately be several: OpenCL, CUDA, plus various CPU backends. (In the future, we plan to add the ability to change the default Accelerate backend either at the <strong>runPar</strong> site, or statically. Stay tuned for that. But for now just use <strong>runAccWith</strong>.)<br />
<br />
The reader might protest that it is possible to use <strong>CUDA.run</strong> directly within a <strong>Par</strong> computation, so why go to the extra trouble? The advantage of using <strong>runAcc</strong> is that it informs the <strong>Par</strong> scheduler of what’s going on. The scheduler can therefore execute other work on the CPU core that would otherwise be waiting for the GPU. An application could achieve the same effect by creating a dedicated thread to talk to the GPU, but that wouldn’t jive well with a pure computation (<strong>forkIO</strong>), and it’s easier to let <strong>meta-par</strong> handle it anyway.<br />
<br />
The second benefit of integrated scheduling is that the scheduler can automatically divide work between the CPU and GPU. Eventually, when there are <a href="https://github.com/HIPERFIT/accelerate-opencl">full-featured, efficient CPU-backends for Accelerate</a>, this will happen transparently. For now you need to use <strong>unsafeHybrid</strong> described in the next section. Finally, our soon-forthcoming <a href="https://github.com/simonmar/monad-par/tree/4332a2dc6fab7ccdb702ad5b285e052f62b43c14/meta-par-dist-tcp">CPU/GPU/Distributed schedulers</a> can make more intelligent decisions if they know where all the calls to GPU computations occur.<br />
<br />
<h2 id="hybrid-cpugpu-workloads.">
Hybrid CPU/GPU workloads.</h2>
The <a href="http://hackage.haskell.org/package/meta-par">meta-par</a> and <a href="http://hackage.haskell.org/package/meta-par-accelerate">meta-par-accelerate</a> packages, as currently released, include a generalized work-stealing infrastructure. The relevant point for our purposes here, is that the CPU and GPU can each steal work from one another. Work-stealing is by no means the most sophisticated CPU/GPU partitioning on the scene. Much literature has been written on the subject, and it can get quite sophisticated (for example, modeling memory transfer time). However, as on regular multicores, work-stealing provides an admirable combination of simplicity and efficacy. For example, if a given program runs much better on the CPU or the GPU, respectively, then that device will end up doing more of the work.<br />
<br />
In the current release, we use <a href="http://www.cs.indiana.edu/~rrnewton/haddock/abstract-par-accelerate/Control-Monad-Par-Accelerate.html#v:unsafeHybrid">unsafeHybridWith, documented here</a>, to spawn a task with two separate implementations—one CPU and one GPU—leaving the scheduler to choose between them. Here’s a silly example:<br />
<pre class="sourceCode literate haskell"><code class="sourceCode haskell"><span class="ot">hybrid ::</span> <span class="dt">Par</span> (<span class="dt">IVar</span> <span class="dt">Float</span>)
hybrid <span class="fu">=</span> unsafeHybridWith CUDA.run (<span class="ot">`indexArray`</span> <span class="dt">Z</span>)
</code><span style="background-color: transparent;"> (</span><span class="fu">return</span><span style="background-color: transparent;"> </span><span class="fl">33.0</span><span style="background-color: transparent;">, triv)</span></pre>
<pre class="sourceCode literate haskell"><code class="sourceCode haskell">runHybrid <span class="fu">=</span> <span class="fu">print</span> (runPar (hybrid <span class="fu">>>=</span> get))</code></pre>
The call to <strong>unsafeHybridWith</strong> is passed a task that consists of a separate CPU <strong>(return 33.0)</strong> and GPU (<strong>triv</strong>) component. We must guarantee that the two computations yield the same result (hence the <strong>"unsafe"</strong> moniker). The <em>indexArray</em> bit is a conversion function that converts the result from the GPU computation to a type that matches the result from the CPU alternative.<br />
<br />
Typically, you would write a recursive algorithm like the following:<br />
<pre class="sourceCode literate haskell"><code class="sourceCode haskell"><span class="co">-- | Recursive mergesort that bottoms out to CPU or GPU:</span>
<span class="ot">parSort ::</span> <span class="dt">Vector</span> <span class="dt">Int</span> <span class="ot">-></span> <span class="dt">Par</span> (<span class="dt">Vector</span> <span class="dt">Int</span>)
parSort vec <span class="fu">=</span>
<span class="kw">if</span> <span class="fu">length</span> vec <span class="fu"><=</span> threshold
<span class="kw">then</span> unsafeHybridWith CUDA.run <span class="fu">undefined</span>
(cpuSort vec, gpuSort vec) <span class="fu">>>=</span> get
<span class="kw">else</span> <span class="kw">let</span> n <span class="fu">=</span> (<span class="fu">length</span> vec) <span class="ot">`div`</span> <span class="dv">2</span>
(l, r) <span class="fu">=</span> <span class="fu">splitAt</span> n vec
<span class="kw">in</span> <span class="kw">do</span> lf <span class="ot"><-</span> spawn_ (parSort l)
r' <span class="ot"><-</span> parSort r
l' <span class="ot"><-</span> get lf
parMerge l' r'</code></pre>
<pre class="sourceCode literate haskell"><code class="sourceCode haskell"><span class="ot">parMerge ::</span> <span class="dt">Vector</span> <span class="dt">Int</span> <span class="ot">-></span> <span class="dt">Vector</span> <span class="dt">Int</span> <span class="ot">-></span> <span class="dt">Par</span> (<span class="dt">Vector</span> <span class="dt">Int</span>)
<span class="ot">cpuSort ::</span> <span class="dt">Vector</span> <span class="dt">Int</span> <span class="ot">-></span> <span class="dt">Par</span> (<span class="dt">Vector</span> <span class="dt">Int</span>)
<span class="ot">gpuSort ::</span> <span class="dt">Vector</span> <span class="dt">Int</span> <span class="ot">-></span> <span class="dt">Acc</span> (<span class="dt">Array</span> <span class="dt">DIM1</span> <span class="dt">Int</span>)
<span class="ot">threshold ::</span> <span class="dt">Int</span></code></pre>
The user of this API still has to make some tough decisions. Setting the threshold determines the granularity at which work will be farmed out between the CPU and GPU. To achieve effective load balance, the problem must be subdivided into sufficiently small pieces to deal with the slower rate at which the CPU may complete work items. Further, in this particular example the GPU may not have enough memory to process the entire array, meaning that the problem must be subdivided at least until it fits in memory.<br />
<br />
On the other hand, subdividing the problem further means more round trips through the GPU, more driver calls, synchronization. In this particular example the total volume of data transferred is the same (it’s linear), but for other algorithms that may not be the case.<br />
<br />
<h2 id="notes-for-hackers">
Notes for Hackers</h2>
If you want to work with the github repositories, you need to have GHC 7.4 and the latest cabal-install (0.14.0). You can check everything out here:<br />
<pre><code>git clone git://github.com/simonmar/monad-par.git --recursive</code></pre>
Try <strong>make mega-install-gpu</strong> if you already have CUDA installed on your machine. Explore the README’s <a href="https://github.com/simonmar/monad-par/blob/master/README.md">here</a> and <a href="https://github.com/simonmar/monad-par/blob/master/examples/README.md">here</a> for more information.<br />
<br />
<h2 id="notes-for-hackers">
Nightly Testing</h2>
The GPU-programming portions of meta-par require the latest and greatest GHC (7.4). <a href="http://tester-lin.soic.indiana.edu:8080/job/monad-par_master_741/">Their nightly testing results are on a Jenkins instance here.</a> The Jenkins setup tests Linux and Windows. Mac OS we test regularly on our laptops, but don’t have a Jenkins instance for yet.<br />
<br />
The core <a href="http://hackage.haskell.org/package/meta-par">meta-par</a> package and the original <a href="http://hackage.haskell.org/package/monad-par">monad-par</a> package do not depend on any recent GHC additions. <a href="http://tester-lin.soic.indiana.edu:8080/job/monad-par_master_oldversions/">Nightly testing results for those packages, on GHC 6.12, 7.0, and 7.2, can be found here.</a><br />
<br />
<h2 id="appendix-documentation-links">
Appendix: Documentation Links</h2>
Because Hackage can’t always build documentation for packages with foreign dependencies, here are some documentation links for the relevant packages:<br />
<ul>
<li><a href="http://www.cs.indiana.edu/~rrnewton/haddock/abstract-par/">abstract-par</a></li>
<li><a href="http://www.cs.indiana.edu/~rrnewton/haddock/abstract-par-accelerate">abstract-par-accelerate</a></li>
<li><a href="http://www.cs.indiana.edu/~rrnewton/haddock/meta-par/">meta-par</a></li>
<li><a href="http://www.cs.indiana.edu/~rrnewton/haddock/meta-par-accelerate/">meta-par-accelerate</a></li>
<li><a href="http://www.cs.indiana.edu/~rrnewton/haddock/accelerate/">accelerate</a></li>
<li><a href="http://www.cs.indiana.edu/~rrnewton/haddock/accelerate-io/">accelerate-io</a></li>
<li><a href="http://www.cs.indiana.edu/~rrnewton/haddock/accelerate-cuda/">accelerate-cuda</a></li>
</ul>
<h2 id="end">
</h2>Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com2tag:blogger.com,1999:blog-2635445209720689121.post-17223101259295842622012-04-26T08:46:00.000-07:002012-04-26T08:46:01.000-07:00Dropbox wiki gone -- Why we little people must Clone the CloudClone the cloud. It's better for everyone. <br />
<br />
With github or with Google Drive this happens transparently. The user has a full copy of what's on the server. Github <a href="https://github.com/blog/699-making-github-more-open-git-backed-wikis">even does this for wiki and web page data</a>, which is great, and some people<a href="http://blog.envylabs.com/2009/08/publishing-a-blog-with-github-pages-and-jekyll/"> create static HTML blogs using github</a> and tools like <a href="https://github.com/mojombo/jekyll/wiki">Jekyll</a>.<br />
<br />
<br />
But unfortunately, most text that's poured into comments, forums, wikis, and blogs seems to have a dubious shelf life. The case in point is that Dropbox took down their wiki a while back, making <a href="http://wiki.dropbox.com/Symlink%20Semantics%20and%20Workarounds">this link</a> in <a href="http://parfunk.blogspot.com/2011/02/dropbox-semantics-oh-that-there-were.html">one of my previous posts</a> dead. Poof. In this particular case I kept a copy in the text file I pasted it from, so I am republishing it below.<br />
<div>
<br /></div>
<div>
"Data liberation" doesn't quite fix the problem, either. Some companies are better than others about providing optional downloads of your data. Google, in particular, is very good. But laborious opt-in downloads (that most people don't use) aren't good enough. Always-on synchronization or mirroring is called for.</div>
<div>
<br /></div>
<div>
I'm optimistic. With the mass movement towards synchronization based approaches (SkyDrive, iCloud, Google Drive, etc) and VCS (github, bitbucket) I hope we are moving in the right direction. And why not? Everyone wins: </div>
<div>
<ul>
<li>For the cloud service provider it means an extra backup copy that reduces the risk of angry customers if a software failure (or more likely, account security breach) destroys data. It also provides local caching, which always makes good engineering sense.</li>
<li>For the user, you get to access your data on the plane and know that it will live on when the company in question decides to retire the cloud service you've been using.</li>
</ul>
<div>
<br /></div>
</div>
<br />
<h2 id="current-behavior">
<a name='more'></a></h2>
<h2 id="current-behavior">
Reproduced Dropbox Wiki page on Symlinks</h2>
<h2 id="current-behavior">
Current Behavior</h2>
Dropbox Mac and Linux clients currently <em>follow</em> symbolic links. This means that the links turn into directories on other Dropbox clients (but remain as symbolic links on the original client). This behavior can be useful for bringing directories into your Dropbox, but can cause serious problems when the source and destination of the link are both within Dropbox. In this scenario the data becomes <em>duplicated</em> on other hosts and the subsequent semantics of the arrangement are unclear and perhaps unspecified.<br />
Consider a folder <code>foo</code> inside your Dropbox, and a symbolic link <code>bar</code> pointing to it.<br />
<pre><code>
</code></pre>
<pre><code>Dropbox/
foo/
bar -> foo
</code></pre>
<br />
Let's say we introduce <code>foo</code> and <code>bar</code> on our Mac laptop, let it sync, and then switch to our Linux desktop. On the Linux machine, foo and bar appear to be completely separate directories. Indeed, if we put the Mac laptop to sleep, these directories remain independent. We can then make changes on the Linux desktop. But when the Mac laptop reconnects it of course still has <code>bar</code> as a symbolic link, and views the directories as one. It will (hopefully) serve as a tunnel between the two directories, reconciling the changes that were made to each. Then the reconciled directories are pushed back to the server and appear identical again on the Linux desktop.<br />
<br />
Note that if there are conflicts, how those changes are applied may depend on the order that the Dropbox client happens to sync <code>foo</code> vs. <code>bar</code> on the Mac client. Yet conflicts are the bread and butter of synchronization solutions so this is not unexpected.<br />
<br />
This "delayed mirroring" semantics may be acceptable, but as of this writing (Feb 2011) it is implemented inconsistently. If both clients are online and a new file is introduced within <code>foo</code> or <code>bar</code> on the Linux client, it may or may not appear in its counterpart directory. Probably the use of <a href="http://en.wikipedia.org/wiki/Inotify">file system monitoring</a> is responsible for the discrepancies, and forcing dropbox to restart (and re-index) will fix transient problems.<br />
<table>
<thead>
<tr class="header">
<th align="left"><br />
A Workaround</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
Of course you can always simply avoid putting folders with symlinks in Dropbox, but if you must open this pandora's box, then you can either live with the situation described above or you can go around Dropbox and try to <em>synchronize the symlinks yourself</em>. Fortunately, you can do this because Dropbox does not damage symlinking arrangements on a particular client once they exist.<br />
Usually, a path's status as symlink or folder does not change often, so it is sufficient to perform a non-dropbox synchronization only occasionally and allow Dropbox to perform the day to day synchronization. One option for manually synchronizing directories that respects symbolic links is the <a href="http://www.cis.upenn.edu/~bcpierce/unison/">Unison file synchronizer</a>.<br />
<h2 id="possible-future-solutions-to-consider">
Possible Future Solutions to Consider</h2>
A votebox entry on symbolic links <a href="https://www.dropbox.com/votebox/350/preserve-implement-symlink-behaviour">can be found here</a>.<br />
<br />
One proposal would be a hybrid approach --- to follow symbolic links that point <em>outside</em> of the dropbox folder but preserve those that are inside.<br />
<br />
The problem with Windows support remains a major one. (Again, Unison can serve as a point of reference here.) But even if internal symbolic links inside a Dropbox are unusable on Windows clients, this may be preferable to the correctness and data integrity problems that can result from the current behavior.<br />
<br />Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com5tag:blogger.com,1999:blog-2635445209720689121.post-21826207229387659032012-03-26T12:31:00.011-07:002012-03-27T06:28:02.591-07:00Potential GSOC: Tweak memory-reuse analysis tools for GHC compatibility<div><span>Some program instrumentation and analysis tools are language agnostic. Pin and Valgrind use binary rewriting to instrument an x86 binary on the fly and thus in theory could be used just as well for a Haskell binary as for one compiled by C. Indeed, if you download Pin from</span><span style="font-family: Georgia, serif; "> </span><a href="http://pintool.org/" style="font-family: Georgia, serif; ">pintool.org</a><span style="font-family: Georgia, serif; ">, you can use the included open source tools to immediately begin analyzing properties of Haskell workloads -- for example the total instruction mix during execution.</span></div><div><span><br /></span></div><div><span>The problem is that aggregate data for an entire execution is rather coarse. It's not correlated temporally with phases of program execution, nor are specific measured phenomena related to anything in the Haskell source.</span></div><div><span><br /></span></div><div><span>This could be improved. A simple example would be to measure memory-reuse distance (an architecture-independent characterization of locality) but to distinguish <b>garbage collection</b> from <b>normal memory access</b>. It would be quite nice to see a histogram of reuse-distances in which GC accesses appear as a separate layer (different color) from normal accesses.</span></div><div><span><br /></span></div><div><span>How to go about this? Fortunately, the existing <a href="http://boegel.kejo.be/ELIS/mica/">MICA pintool</a> can build today (v0.4) and measure memory reuse distances. In fact, it already produces per-phase measurements where phases are delimited by dynamic instruction counts (i.e. every 100M instructions). All that remains is to tweak that definition of phase to transition when GC switches on or off.</span></div><div><span><br /></span></div><div><span>How to do that? Well, Pin <a href="http://www.cs.virginia.edu/kim/publicity/pin/docs/45467/Pin/html/group__RTN__BASIC__API.html#g8622a6ba858eb8d55df4e006eb165e57">has existing methods for targeted instrumentation of specific C functions</a>. By targeting appropriate functions in the GHC RTS, this analysis tool could probably work <i>without requiring any GHC modification at all</i>.</span></div><div><span><br /></span></div><div><span>A further out goal would be to correlate events observed by the binary rewriting tool and those recorded by GHC's traceEvent.</span></div><div><span><br /></span></div><div><span>Finally, as it turns out this would NOT be the first crossing of paths between GHC and binary rewriting. Julian Seward worked on GHC before developing valgrind:</span></div><div><span><a href="http://www.techrepublic.com/article/open-source-awards-2004-julian-seward-for-valgrind/5136747"> http://www.techrepublic.com/article/open-source-awards-2004-julian-seward-for-valgrind/5136747</a></span></div><div><span><br /></span></div><div>[<a href="http://hackage.haskell.org/trac/summer-of-code/ticket/1619">Note this proposal is also posted on the GSOC trac.</a>]</div><div><span><br /></span></div>Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0tag:blogger.com,1999:blog-2635445209720689121.post-4627292389030145972012-02-08T08:56:00.001-08:002012-02-16T07:06:52.732-08:00Potential GSoC: Haskell Lock-free Data Structure ImplementationsThe GHC Haskell compiler recently gained the capability to generate atomic compare-and-swap (CAS) assembly instructions. This opens up a new world of data-structure implementation possibilities. <div><br />Furthermore, it's an important time for concurrent data structures. Not only is the need great, but the design of concurrent data structures has been a very active area in recent years, as summarized well by Nir Shavit <a href="http://cacm.acm.org/magazines/2011/3/105308-data-structures-in-the-multicore-age/">in this article.</a> <br /><br />Because Haskell objects containing pointers can't efficiently be stored outside the Haskell heap, it is necessary to reimplement these data structures for Haskell, rather than use the FFI to access external implementations. There are already a couple of data structures implemented in the following library (queues and deques) :<br /><div><br /></div><div> <a href="https://github.com/rrnewton/haskell-lockfree-queue">https://github.com/rrnewton/haskell-lockfree-queue</a></div><div><br /></div><div>But, this leaves many others, such as:</div><div><ul><li>Concurrent Bags</li><li>Concurrent Hashtables</li><li>Concurrent Priority Queues<br /></li></ul><div>A good point of reference would be the <a href="http://libcds.sourceforge.net/">libcds collection of concurrent data structures for C++</a> (or those that come with Java or .NET):</div><div><p>One of the things that makes implementing these data structures fun is that they have very short algorithmic descriptions but a high density of thought-provoking complexity.</p></div><p>A good GSoC project would be to implement 2-3 data structures from the literature, and benchmark them against libcds implementations.</p><p>UPDATE: <a href="http://hackage.haskell.org/trac/summer-of-code/ticket/1608">Here's a link to the Trac ticket.</a></p><p><br /></p><p><br /></p><br /><br /></div></div>Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com2tag:blogger.com,1999:blog-2635445209720689121.post-58943496094579980042011-12-27T21:29:00.000-08:002011-12-27T23:18:14.977-08:00Wuala, SpiderOak, and Dropbox: Feature Summary and a Little Testing<div>Dropbox, SpiderOak, and Wuala seem to be the current contenders for someone who wants synchronized cloud storage together with Linux/Mac support. I've tried all three and come up with a few observations that may be of use.</div><div><br /></div><div>First I did a little test of Dropbox, SpiderOak, and Wuala. I wrote to a text file on one machine inside my LAN and timed how long it took for the file to appear on the other machine. It took 5, 24, and 68 seconds respectively.</div><div><br /></div><div>These times are not surprising given that Dropbox implements a "LAN sync" capability, whereas SpiderOak goes through the central server, and Wuala not only goes through the server (in Europe; I'm in Indiana) it further <b>does not use inotify</b>, i.e. it has to periodically scan the file system for changes.</div><div><br /></div><div>But there are some major reasons to chose SpiderOak or Wuala over Dropbox. One of them is that both SpiderOak and Wuala have<b> client-side encryption</b> such that their employees shouldn't be able to access your files. </div><div><br /></div><div>Further, there's the handling of symbolic links, <a href="http://parfunk.blogspot.com/2011/02/dropbox-another-way-to-shoot-your-foot.html">which I have complained about before</a> ***. Wuala syncs them. SpiderOak ignores them (which is still better than Dropbox, which follows them resulting in weird behavior).</div><div><br /></div><div>Wuala also has some other unique desirable features as well as some major technical drawbacks. </div><div><ol><li>Wuala <b>used to</b> allow a P2P mode in which users traded space on their own machines to become part of the storage network. A very neat concept and a good way to get a lot of cloud storage at a reasonable price. (By contrast, 3TB of storage on Amazon, to match one external HD, <a href="http://aws.amazon.com/s3/pricing/">is $5000 a year</a>).</li><li>Wuala has a time travel view that lets you see the previous state of an entire folder. Dropbox and SpiderOak have a <b>single file fixation</b>. You can view previous versions of an individual file using their respective UIs. This is great for, say, a word document, but very poor for a directory containing code or multiple latex files.</li><li>FUSE support. Wuala allows the remote drive to be directly mounted via FUSE without requiring everything to be sync'd locally. In theory this would seem to combine the benefits of something like Dropbox with traditional network file systems like NFS and AFS.</li></ol></div><div>And then the drawbacks. Unfortunately these are drawbacks that strike at the core -- the bread and butter of syncing. First, as mentioned above, Wuala doesn't use inotify for to allow the OS to alert it of changed files. Second, Wuala doesn't allow exclusion of files based on name or extension -- a major drawback it shares with Dropbox which makes it inefficient to rebuild compiled projects inside a synced folder. (Note: Wuala also used to not support incremental/differential uploads. That has been implemented.) </div><div><br /></div><div>In summary:</div><div><br /></div><div>Wuala </div><div><ul><li>+ Syncs symbolic links</li><li>+++ Time travel view</li><li>+ FUSE-mount remote drive without syncing</li><li>- No<a href="http://bugs.wuala.com/view.php?id=5070"> inotify support yet</a></li><li>- No<a href="http://bugs.wuala.com/view.php?id=2428"> exclusion of files from sync yet</a></li><li>- No <a href="https://forum.wuala.com/viewtopic.php?f=15&t=744">systray icon / background mode under mac os yet</a></li><li>- [US] Data centers in Europe</li></ul><div>SpiderOak</div></div><div><ul><li>+ Ignores symbolic links (rather than doing something terrible)</li><li>+ Exclude files from sync via extension or regex</li></ul></div><div>Dropbox</div><div><ul><li>+ LAN sync</li><li>? More geographic coverage (amazon)</li></ul></div><div><br /></div><div>*** P.S. In an ironic act of data non-preservation it looks like the work-arounds I'd posted to the dropbox wiki (http://wiki.dropbox.com/Symlink%20Semantics%20and%20Workarounds) were lost because they took down the entire wiki.</div><div><br /></div><div> </div><div><br /></div>Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com3tag:blogger.com,1999:blog-2635445209720689121.post-20654779544295781112011-02-24T19:27:00.001-08:002012-04-26T09:25:00.334-07:00Dropbox - another way to shoot your foot<div>
Whew, the data loss in the last post actually wasn't that bad because the folder was version controlled anyway (and stored on another server). I <i>thought</i> only using dropbox for working copies would keep me safe.</div>
<div>
<br /></div>
So much for that idea! In a sleepy state just now I managed to get Dropbox to delete scattered files <b>outside of</b> my Dropbox folder! <br />
<div>
<br /></div>
<div>
This comes back to the <a href="http://parfunk.blogspot.com/2012/04/dropbox-wiki-gone-why-we-little-people.html">symlink semantics</a>, but it can bite you in strange ways. Here's how you can really screw yourself. First, let's say you've got a working copy that stores all your notes and other reference text files that you search often:</div>
<div>
<div>
<div>
<pre> mv foobar ~/dropbox/foobar</pre>
</div>
<div>
Let's further suppose that you are on a Mac or Linux client and <span class="Apple-style-span">foobar</span> contains symbolic links that point to other scattered text files elsewhere in your personal archives that you may want to have at hand (for grepping). </div>
<div>
<br /></div>
<div>
Next, you want to make a major change to <span class="Apple-style-span">foobar</span>, so you back it up, just to be safe:</div>
<div>
<pre> cp -a ~/dropbox/foobar ~/dropbox/foobar_bak</pre>
</div>
<div>
A while later you come across this backup (on a different dropbox-linked machine) and delete it:</div>
<div>
<br /></div>
<div>
<span class="Apple-style-span" style="font-family: monospace; font-size: 14px; white-space: pre;"> rm -rf foobar_bak</span></div>
<div>
<br /></div>
<div>
BAM! That's it, you just screwed yourself. </div>
<div>
<br /></div>
<div>
How? When Dropbox went back to your original Mac/Linux machine to delete <span class="Apple-style-span">foobar_bak</span>, it decided to FOLLOW THE SYMLINKS. That's right, it deleted the original files. Even though the original <span class="Apple-style-span">foobar</span> directory is still there, its links are now broken of course.</div>
<div>
<br /></div>
<div>
The whole point of these links was organizational. They pointed all over the place. Even if you have backups you now have to track them down and restore those scattered files. (Which is what I just spent my time doing.) I guess I'm a synchronization masochist because I seem to ask for this kind of punishment.<br />
<div>
<br /></div>
<div>
Bottom line. If you're a power user, avoid Dropbox or keep your usage very light, e.g. stick an office document in there every once in a while. Or be ready to suffer.</div>
</div>
</div>
</div>Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0tag:blogger.com,1999:blog-2635445209720689121.post-55987748020877945892011-02-24T19:26:00.000-08:002011-02-25T17:23:08.262-08:00A New Dropbox Nightmare<div style="text-align: center;"><br /></div>I <i>was</i> cautiously optimistic and hopeful -- "maybe Dropbox is the synchronization solution I've been waiting for" -- but that sentiment is quickly being replaced with <i>jaded</i>.<div><br /></div><div>First, it is good to recognize that backup and synchronization (mirroring) are orthogonal. Mirroring only insures that mistakes get propagated. Well, I just experienced my second Dropbox <b>spontaneous deletion event</b>, which, due to the miracle of synchronization, was applied to all of my computers and to the cloud.</div><div style="text-align: center;"><br /></div><div>What can be worse than a silent deletion event for a service like Dropbox? Nothing that I can think of. I can't babysit the 96,000 files in my Dropbox to catch something like this.</div><div style="text-align: center;"><br /></div><div>Now the details. I came across a directory that had 95% of the files deleted (leaving a random few remaining). Scrolling back through the event history I found that this happened after linking a new client, which happened to be a Windows virtual machine. Below is a snippet of the event audit trail. The entire series of events happening at 10:09 was spurious (including deleting 411 files and 206 folders).</div><div><pre><span class="Apple-style-span"><span class="Apple-style-span">You edited the file .saves-25811-rnmacbook13~. 11:27 AM<br />You renamed the file intel_aes.c to test038.h.ERR. 10:09 AM<br />You renamed the file setup to test047.h.ERR. 10:09 AM<br />You added 686c9fad58174636354 and 78 more files. 10:09 AM<br />You edited simple_cycle.out and 49 more files. 10:09 AM<br />You deleted 6c88b2f3aaaa585db12 and 411 more files. 10:09 AM<br />You added eb and 6 more folders. 10:09 AM<br />You deleted intel_aes_lib and 206 more folders. 10:09 AM<br />You moved 04 and 24 more folders. 10:09 AM<br />You added the folder fabl. 12:42 AM<br />The computer rn-waspvm was linked to your account 12:24 AM<br />You edited the file ghc_setup.txt. 12:24 AM</span><br /></span></pre></div><div>A couple additional oddities show up in the above:</div><div><ul><li>it waited over 9 hours after the new client was linked to go crazy</li><li>Files were "'renamed", <span class="Apple-style-span" style="font-family: monospace; font-size: 15px; white-space: pre; ">intel_aes.c to test038.h.ERR</span> Huh? Is that a remove being aliased with an add to create a fake rename? Possibly a secondary bug.</li></ul></div><div>I went back and checked out the state of that Windows client just now. A clue! The only files it successfully downloaded were the ones <b>not</b> deleted in the above nonsense. So Dropbox confused a partially downloaded directory with a new version of the directory (deleting most of the contents). Unison has never done anything like that to me in ten years of heavy use!</div><div><br /></div><div>It may be corroboration that the Windows client currently thinks it "hasn't connected yet" [Screenshot below]. Oh really?</div><div><br /></div><div><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5eBRrpPQE5hdDNUALZu4-BG26kIirhYignWWI9uzcwXtU_AJwxV3LyOlfJEi1gUKutOIirK-MtKzyTHhNYpzpp6-PyOGlVNhXOcjSSby_1uOqI08SgoHEy_eymhnc5ic2BRVp4HzipXPc/s320/virtualbox_DROPBOX_SAYS_IT_NEVER_SYNCED.png" style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 299px;" border="0" alt="" id="BLOGGER_PHOTO_ID_5577489507992823826" /></div><div><br /></div>Well, perhaps this is "fool me twice, shame on me". I should have known better -- my wife tried Dropbox about a year ago, dumped her ~50 gigabyte personal archive in all at once, then <i>moved </i>a folder, and ended up with her data in a never-fully-sychronized, confused state. It still hasn't been completely fixed; there's a backup, but also new stuff mixed into the messed up Dropbox version. I chalked that one up to bad behavior while under load and before the first sync is successful (plus the dangers of syncing being <i>too</i> implicit and automatic). Still, I wanted to give Dropbox another chance; trying it myself this time, thinking that: <div><div><div><div><ul><li>It's surely improving and ...</li><li>I would be more cautious, spoon-feed it, and pay more attention to its syncing state. </li></ul>But clearly that didn't save me.</div><div><br /></div></div></div></div>Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0tag:blogger.com,1999:blog-2635445209720689121.post-30163400361982504222011-02-15T10:13:00.000-08:002014-08-12T20:18:54.602-07:00Dropbox semantics - oh that there were such a thing<div>
After some recent Dropbox problems I've been having (not the first unfortunately) I added an entry on the Dropbox wiki explaining what happens if you are so impertinent as to add directories containing symbolic links to your Dropbox folder:</div>
<div>
<br /></div>
<div>
<a href="http://wiki.dropbox.com/Symlink%20Semantics%20and%20Workarounds">http://wiki.dropbox.com/Symlink%20Semantics%20and%20Workarounds</a></div>
<div>
<br />
[UPDATE - dropbox took down their wiki, but <a href="http://parfunk.blogspot.com/2012/04/dropbox-wiki-gone-why-we-little-people.html">I reposted the page here</a>.]<br />
<br /></div>
<div>
One day maybe we'll have something with the correctness and robustness of <a href="http://www.cis.upenn.edu/~bcpierce/unison/">Unison</a> and the convenience of Dropbox. People will dance in the streets. I think Dropbox can be that solution if they work at it.</div>
<div>
<br />
UPDATE - these issues are explained at greater length in this post:<br />
<br />
<a href="http://aurelio.net/articles/dropbox-symlinks.html">http://aurelio.net/articles/dropbox-symlinks.html</a><br />
<br /></div>
Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com1tag:blogger.com,1999:blog-2635445209720689121.post-67588367183295604022010-10-14T19:55:00.000-07:002010-10-14T20:50:56.413-07:00Hacking together a working version of Haskell Platform 2010.2 for GHC 7.1I don't know if GHC developers typically attempt to bring up a full, cabal-install based setup around development versions of the compiler, but that's what I wanted. To that end I hacked up a modified version of <a href="http://hackage.haskell.org/platform/changelog.html">Haskell Platform 2010.2</a> that works with GHC 7.1.20101014. This is a recent version of GHC that includes, among other things:<br /><ul><li>Major run time improvements: namely, a BLACKHOLE fix that greatly improve parallel performance in some cases, relative to 6.12.</li><li>The LLVM backend.</li><li>The new type inference engine.</li></ul>The modified version of Haskell Platform can be downloaded from the link below. You should be able to build it against a recent GHC that you can grab <a href="http://www.haskell.org/ghc/dist/current/dist/">from the nightly snapshots</a> (try 2010.10.14 to be safe). If that doesn't work, you can download my own build of the compiler below (Linux 64 bit) which is inflexible and needs to be unpacked in <span style="font-family:courier new;">/opt/</span>. I'm including links to download it with or without the platform pre-installed:<br /><ul><li>Download (2MB): <a href="http://people.csail.mit.edu/newton/files/haskell-platform-2010.2-HACKED.tar.bz2">haskell-platform-2010.2-HACKED.tar.bz2</a><br /></li><li>Download (100MB): <a href="http://people.csail.mit.edu/newton/files/ghc7.1.20101014.tar.bz2">ghc7.1.20101014.tar.bz2</a></li><li>Download (112MB): <a href="http://people.csail.mit.edu/newton/files/ghc7.1_with_platform_preeinstalled.tar.bz2">ghc7.1_with_platform_preeinstalled.tar.bz2</a><br /></li></ul>After that you should be ready for installing other software via "cabal install".<br /><br />The modified haskell platform tarball contains my notes on the hacks that were necessary, copied below:<br /><br /><hr /><pre><br />I did a few hacks to get this working with GHC 7.1.20101014<br /><br />First was to upgrade the MTL package to a slightly newer version,<br />downloaded from Hackage. Ditto for deepseq.<br /><br />The second was to hack the .cabal file in haskell-src to be less<br />picky about upperbounds on version numbers.<br /><br />But I still ran into compile problems with haskell-src. (And there's<br />no new version of it released at the moment to upgrade to.) Namely, I<br />got the following compile error:<br /><br />Language/Haskell/Syntax.hs:67:8:<br />Could not find module `Data.Generics.Instances':<br />Use -v to see a list of the files searched for.<br /><br />Ok, I included "syb" in the list of packages.<br /><br />Next, build error on quickcheck... upgraded to 2.3.0.2<br />But that didn't fix the problem -- "split" is still undefined.<br /><br />The package 'parallel' had to be loosened up to tolerate newer containers versions.<br /><br />HTTP complained about wanting base 3... but why was the "old-base"<br />flag set anyway?<br /><br />Ditto zlib.<br /><br />Finally, the cabal-install package also required some relaxation of<br />version numbers, and worse it seems like the type of a library<br />function has changed from a Maybe result to a definite one.<br /><br />Here I had to make a choice between updating to cabal-install 0.9.2<br />or hacking the 0.8.2 version. It works to add an "fmap Just" to get<br />0.8.2 to build, and besides the 0.9.2 version I have is actually just<br />the darcs head -- there hasn't been a release yet.<br /><br />After picking the hacked 0.8.2 version, `make` succeeded for my<br />modified haskell-platform-2010.2.<br /></pre>Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0tag:blogger.com,1999:blog-2635445209720689121.post-75043728191634965432010-06-29T11:19:00.001-07:002010-06-29T11:46:17.330-07:00Good ol' FP infightingA Schemer by upbringing, I know all about infighting!<br /><br />It seems that there's been an ongoing back and forth between certain ML and Haskell partisans about performance predictability. Further, there's <a href="http://flyingfrogblog.blogspot.com/2009/04/f-vs-ocaml-vs-haskell-hash-table.html">been the accusation that Haskell has horrible hash table performance</a>. (Haven't ran these tests myself yet, but I'd like to.)<br /><br />Anyway, the <a href="http://hackage.haskell.org/package/haskell-cnc">Haskell CnC</a> project that I announced on the Intel blogs has been garnering some feedback. It got panned particularly brutally <a href="http://flyingfrogblog.blogspot.com/2010/06/intel-concurrent-collections-for.html">by Jon Harrop here</a>. I don't know, maybe that's just what Jon likes to do. In any case, I wrote a response in the comments and it grew big enough that I thought I'd make it into a post (below).<br /><br /><hr /><br /><br /><span style="font-weight:bold;">Response to: <a href="http://flyingfrogblog.blogspot.com/2010/06/intel-concurrent-collections-for.html">http://flyingfrogblog.blogspot.com/2010/06/intel-concurrent-collections-for.html</a></span><br /><br />I think there's a bit of a misunderstanding here in the initial conflation of CnC with "graph reduction". CnC is a programming model based on static graphs (no reduction) like synchronous data flow. (I recommend <a href="http://groups.csail.mit.edu/cag/streamit/">the papers on StreamIT in this area</a>.)<br /><br />If anything, you could view this as <em>backing down</em> from some of the traditional Haskell implementation decisions. The user makes decisions as to the granularity of the parallel tasks (the granularity of the nodes in the graph), and CnC eschews lazy evaluation during graph execution. <br /><br />I guess I feel like I've been caught in the middle of some existing flame wars wrt hash tables and performance predictability. I'm aware of the existing battle over Haskell hash tables. I have no problem with hash tables. The Haskell CnC implementation makes no use of persistence and can use mutable map data structures just as well as immutable. I use Haskell hash tables, but as they don't support concurrent update (and aren't high performance to start with) they're not the default. I'd love better hash tables. In other implementations we use TBB hashmaps.<br /><br />Regarding performance predictability -- sure, it's a problem, both because of lazy evaluation and dependence on high level (read fragile) optimizations. But I don't see why this needs to be an argument. People who need great predictability today can use something else. Haskellers should continue to try to improve predictability and scaling. What else can they do? Even in this little CnC project we have made progress -- complex runtime, quirks and all -- and have Haskell CnC benchmarks getting parallel speedups over 20X. Not perfect, but improving. (I'll give your raytracing benchmark a try sometime, if you like; that sounds fun.)<br /><br />I also don't think it's a good idea to endorse Dmitriy Vyukov "kindly" dismissing all of Haskell and Erlang. This is an even more extreme position than Jon's own "doesn't scale beyond a few cores" position.<br /><br />Jon, I'm amenable to your arguments and not necessarily in a different camp (I'm an eager functional programmer more than a lazy one), but I would appreciate not being so cursorily panned!<br /><br />P.S. (Regarding "the only problem that we've solved" being mandelbrot. Indeed that would be bad if it were so. <a href="http://software.intel.com/en-us/blogs/2010/06/24/haskell-cnc-new-paper-available-tests-on-32-and-48-cores/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+IntelSoftwareNetworkBlog+(Intel+Software+Network+Blog)">Please see the paper.</a> We don't have nearly as many benchmarks as I'd like yet, but we're doing Cholesky factorization, the Black-Scholes equation, and the N-body problem at least. Also the paper has some more in-depth discussions about scheduling. None of the current schedulers, by the way, are yet considered by me to be anywhere near an endpoint. Many low hanging fruit remain.) <br /><br />P.P.S We also have a CnC implementation in F#.Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0tag:blogger.com,1999:blog-2635445209720689121.post-47364822901791460222010-06-29T11:05:00.000-07:002010-06-29T11:09:39.414-07:00I guess a blog wouldn't hurt.I've been <span style="font-weight:bold;"><a href="http://software.intel.com/en-us/blogs/author/ryan-newton/">posting a bit on the Intel blogs</a></span> recently. Yet I guess there will always be posts that aren't appropriate in that context, so why not create one of these new fangled Weblogs?Ryan Newtonhttp://www.blogger.com/profile/15916639901554711946noreply@blogger.com0