The Naiad system provides fast incremental and iterative computation for data-parallel workloads
Home Page: http://microsoftresearch.github.io/Naiad/
License: Apache License 2.0
Installation steps
------------------
Naiad is distributed as a Visual Studio 2013 solution, comprising
the following projects:
* "Naiad", which contains the core Naiad system and support libraries,
and builds Microsoft.Research.Naiad.dll. If you create your own project that
uses Naiad, you must add a reference to this library.
* "Lindi", a LINQ-like library for data-parallel iterative computation.
* "DifferentialDataflow", a LINQ-like library supporting incremental
execution of iterative computation.
* "AzureSupport", a library of Naiad components that supports writing to
and reading from Windows Azure Storage.
* "Examples", which contains several example Naiad applications and
builds Examples.exe.
* In addition, the "ClusterSubmission" solution contains experimental support
for launching Naiad programs on Azure and YARN clusters.
These projects can be built and executed using any of the following
implementations of the .NET framework:
* Microsoft .NET Framework 4.5
To build Naiad, follow the steps below that correspond to your setup.
* Building using Visual Studio
1. Open the Naiad.sln file in this directory using Visual Studio.
2. On the menu bar, click Build > Build Solution, or press F6.
* Building using MSBuild (on Windows, without Visual Studio)
1. Ensure that MSBuild is installed on your local machine. You can
get MSBuild by installing the Microsoft .NET Framwork, version
4.5. Typically, the file MSBuild.exe can be found in the path:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe
However, this path may differ, depending on your local
configuration.
2. Open a command prompt, and change to the directory containing the
Naiad.sln file.
3. At the command prompt, run MSBuild.exe.
Getting started with the example programs
-----------------------------------------
As part of this source distribution, the Examples project demonstrates
how to write simple applications using Naiad. After building the Naiad
solution, you will find the executable Examples.exe in the
Examples\bin\Debug directory.
Examples.exe uses command-line arguments to select an example, and it
is recommended that you run it from a command prompt or terminal. For
example, to run the word count example, follow these steps:
1. Open a command prompt or terminal, and change to the
Examples\bin\Debug directory.
2. At the command prompt, type "Examples.exe" to see the usage and
available examples.
3. At the command prompt, type "Examples.exe wordcount" to get started
with the wordcount example.
Known issues with Naiad
-----------------------
This release of Naiad is an alpha release. The amount of testing the code
has been subjected to is limited, mostly to programs we have written. The
email address [email protected] goes to each of the project
members, and is a great place to send questions and comments.
There are several specific issues we know about now, and want to point
out to avoid headaches for everyone involved:
1. In distributed mode, we strongly recommend that each process call
OnNext the same numbers of times before calling OnCompleted.
Several aspects of the graph manager logic consult the epoch numbers
associated with local inputs. Ideally, all process call OnNext the
same number of times; passing no parameters to OnNext causes the
epoch to advance with no input supplied, and is a great way to do this.
2. Naiad currently requires custom datatypes to be structs, rather
than classes. This is not too hard to fix, but does require us
to swing through the code and ensure that every class has an
appropriate default constructor, which all structs have. Structs
also stress the GC a lot less, and are a lot easier to auto-serialize.
Related to this, if you want to use distributed Naiad, the
auto-generated Serialization/Deserialization code requires all of
your fields to be public and mutable (ie: not readonly). This is
because the generated code attempts to set these values on a newly
minted struct, and can’t do this without these settings. More
details on writing custom serialization code in upcoming blog posts.
As Naiad is a research prototype, there are many parts of it that are
likely to change. We’ve tried hard to put interfaces in place that will
remain relatively stable, but there are certainly aspects of the
interfaces we expect will change. Information about programming examples,
the current state of the code, known problems, and upcoming fixes and
features should be available at the project web page:
http://research.microsoft.com/Naiad/,
or at the related blog:
http://bigdataatsvc.wordpress.com/
Contacting us
-------------
If you have questions or comments, please feel welcome to email us at
[email protected]
We'd love to hear your thoughts on the project. Thanks!
The Naiad team.
Crashes with an exception after uploading files:
Exception The cluster version 3.1.0.227 is not supported by this version of the tools. Please upgrade your tools to the newer version. The compatibility range for these tools is 1.6 - 3.0.
System.NotSupportedException: The cluster version 3.1.0.227 is not supported by this version of the tools. Please upgrade your tools to the newer version. The compatibility range for these tools is 1.6 - 3.0.
Server stack trace:
at Microsoft.WindowsAzure.Management.HDInsight.JobSubmission.HDInsightHadoopClient.AssertSupportedVersion(Version hdinsightClusterVersion, IVersionFinderClient versionFinderClient)
at Microsoft.WindowsAzure.Management.HDInsight.JobSubmission.HDInsightHadoopClient.GetJobSubmissionClusterDetails(Boolean ignoreCache)
at Microsoft.WindowsAzure.Management.HDInsight.JobSubmission.HDInsightHadoopClient.<CreateStreamingJobAsync>d__e.MoveNext()
Exception rethrown at [0]:
at Microsoft.WindowsAzure.Management.HDInsight.Framework.Core.Library.TaskExtensions.WaitForResult[T](Task`1 task, TimeSpan timeout)
at Microsoft.WindowsAzure.Management.HDInsight.Framework.Core.Library.TaskExtensions.WaitForResult[T](Task`1 task)
at Microsoft.WindowsAzure.Management.HDInsight.JobSubmission.HDInsightHadoopClient.CreateStreamingJob(StreamingMapReduceJobCreateParameters streamingMapReduceJobCreateParameters)
at Microsoft.Research.Peloponnese.ClusterUtils.AzureYarnClient.Submit(XDocument config, String errorLocation) in d:\Users\misard\src\peloponnese\Utils\ClusterClient.cs:line 386
at Microsoft.Research.Naiad.Cluster.ClusterSubmission.Submit() in c:\gits\naiad\ClusterSubmission\NaiadPeloponneseSupport\ClusterSubmission.cs:line 157
at Microsoft.Research.Naiad.Cluster.Azure.Program.Run(String[] args) in c:\gits\naiad\ClusterSubmission\AzureSubmission\Program.cs:line 153
at Microsoft.Research.Naiad.Cluster.Azure.Program.Main(String[] args) in c:\gits\naiad\ClusterSubmission\AzureSubmission\Program.cs:line 161
Hi Naiad team,
I'm a bit late to the party, but after finally watching the presentation video from SOSP, I was very excited to learn that Naiad is available open source. I'm excited to try it out.
My main development environment is not a windows machine, but I noticed there is a script called "build_mono.sh". I was wondering if you've had any success building it? It might be something else I am doing wrong, but when I try to build using this script, it fails because it cannot find "EventTask", which should be under System.Diagnostics.Tracing. I'm using xbuild version 12.0.0, and Mono 3.10.0.0. As further evidence, the xamarin API doesn't seem to have this class.
Finally, in light of the somewhat recent and very shocking announcement from Microsoft about MSR-SV, do you have any idea what impact this might have on future support for Naiad? I'm very sorry to all of the folks who were impacted.
Thanks,
Alvin
Invoking Examples.exe wordcount -p 2 -n 2 --local with the latest commit (360ff09) causes an ArgumentOutOfRangeException.
The issue was already occurring in release_0.4.
Exception Details:
System.IndexOutOfRangeException was unhandled
HResult=-2146233080
Message=Index was outside the bounds of the array.
Source=Microsoft.Research.Naiad
StackTrace:
at Microsoft.Research.Naiad.BaseController..ctor(Configuration config) in d:\Schuldateien\SS 14\Sources\naiad\Naiad\Runtime\Controlling\Controller.cs:line 771
at Microsoft.Research.Naiad.NewController.FromConfig(Configuration conf) in d:\Schuldateien\SS 14\Sources\naiad\Naiad\Runtime\Controlling\Controller.cs:line 283
at Microsoft.Research.Naiad.InternalOneOffComputation..ctor(Configuration configuration) in d:\Schuldateien\SS 14\Sources\naiad\Naiad\Runtime\SubgraphManager.cs:line 67
at Microsoft.Research.Naiad.NewComputation.FromArgs(String[]& args) in d:\Schuldateien\SS 14\Sources\naiad\Naiad\Runtime\Controlling\Controller.cs:line 220
at Microsoft.Research.Naiad.Examples.WordCount.WordCount.Execute(String[] args) in d:\Schuldateien\SS 14\Sources\naiad\Examples\Naiad\WordCount.cs:line 98
at Microsoft.Research.Naiad.Examples.Program.Main(String[] args) in d:\Schuldateien\SS 14\Sources\naiad\Examples\Program.cs:line 105
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException:
StreamingInputVertex.PerformAction holds a lock on the object while it processes all of its input records (calls to output.Send()). This blocks any threads trying to enqueue work items. It seems that the lock should be able to held for less time, only related to variables/state that might be modified by other threads.
Hi,
The following exception occasionally takes down our computation. We don't really know how to reproduce it in isolation as it happens only when running on large input datasets and after a few minutes of execution. I am not yet sure that our computation is fully deterministic when run multiple times on the same input (we're verifying this now) so the fact that this happens occasionally may be due to non-deterministic behaviour in our code.
"ParsedInputPono" (renamed) is a plain-old-NET-object with a few public fields (of type long, int, string) and a constructor that only copies the arguments to the respective fields.
We're running a debug build on linux with mono (I'll try to rerun with --debug to get line numbers) on about 16 machines with 4 workers each (-t 4).
Can you help us out?
Thank you.
00:07:58.2570129, Graph 0 failed on scheduler 3 with exception:
System.NullReferenceException: Object reference not set to an instance of an object
at Microsoft.Research.Naiad.Serialization.AutoGenerated.ParsedInputPono.TrySerializeMany (Microsoft.Research.Naiad.Serialization.SubArray`1& destination, ArraySegment`1 values) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Serialization.SendBufferPage.WriteElements[ParsedInputPono] (NaiadSerialization`1 serializer, ArraySegment`1 elements) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Serialization.AutoSerializedMessageEncoder`2[ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].WriteElements (ArraySegment`1 records, Int32 srcVertexId) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Serialization.AutoSerializedMessageEncoder`2[ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].Write (ArraySegment`1 records, Int32 srcVertexId) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.Channels.RemoteMailbox`2[ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].Send (Message`2 message, ReturnAddress from) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.Channels.BufferingPostbox`2[ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].FlushBuffer (Int32 index) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.Channels.BufferingPostbox`2[ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].InternalSend (Message`2 records) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.Channels.BufferingPostbox`2[ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].Send (Message`2 records) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.VertexOutputBuffer`2[ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].Send (Message`2 message) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.VertexOutputBufferPerTime`2[ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].SendBuffer () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.VertexOutputBufferPerTime`2[ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].Send (ParsedInputPono record) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Frameworks.Lindi.ExtensionMethods+SelectManyVertex`3[System.String,ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].OnReceive (Message`2 message) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.StandardVertices.UnaryVertex`3[System.String,ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch].<MakeStage>m__0 (Message`2 message, Microsoft.Research.Naiad.Dataflow.StandardVertices.UnaryVertex`3 vertex) [0x00000] in <filename unknown>:0
at (wrapper delegate-invoke) System.Action`2<Microsoft.Research.Naiad.Dataflow.Message`2<string, Microsoft.Research.Naiad.Dataflow.Epoch>, Microsoft.Research.Naiad.Dataflow.StandardVertices.UnaryVertex`3<string, ParsedInputPono, Microsoft.Research.Naiad.Dataflow.Epoch>>:invoke_void_T1_T2 (Microsoft.Research.Naiad.Dataflow.Message`2<string, Microsoft.Research.Naiad.Dataflow.Epoch>,Microsoft.Research.Naiad.Dataflow.StandardVertices.UnaryVertex`3<string, ParsedInputPono, Microsoft.Research.Naiad.Dataflow.Epoch>)
at Microsoft.Research.Naiad.Dataflow.Stage`2+<NewInput>c__AnonStorey2`1+<NewInput>c__AnonStorey3[Microsoft.Research.Naiad.Dataflow.StandardVertices.UnaryVertex`3[System.String,ParsedInputPono,Microsoft.Research.Naiad.Dataflow.Epoch],Microsoft.Research.Naiad.Dataflow.Epoch,System.String].<>m__0 (Message`2 m) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.ActionReceiver`2+<ActionReceiver>c__AnonStorey0[System.String,Microsoft.Research.Naiad.Dataflow.Epoch].<>m__0 (Message`2 m, ReturnAddress u) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.ActionReceiver`2[System.String,Microsoft.Research.Naiad.Dataflow.Epoch].OnReceive (Message`2 message, ReturnAddress from) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.Channels.PipelineChannel`2+Fiber[System.String,Microsoft.Research.Naiad.Dataflow.Epoch].Send (Message`2 records) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.VertexOutputBuffer`2[System.String,Microsoft.Research.Naiad.Dataflow.Epoch].Send (Message`2 message) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.VertexOutputBufferPerTime`2[System.String,Microsoft.Research.Naiad.Dataflow.Epoch].SendBuffer () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.VertexOutputBufferPerTime`2[System.String,Microsoft.Research.Naiad.Dataflow.Epoch].Send (System.String record) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Dataflow.StreamingInputVertex`1[System.String].PerformAction (WorkItem workItem) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Scheduling.Scheduler+WorkItem.Run () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Scheduling.Scheduler.Schedule (WorkItem workItem) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Scheduling.Scheduler.RunWorkItem (Int32 graphId) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Scheduling.Scheduler.RunNotification (Int32 computationIndex) [0x00000] in <filename unknown>:0
Due to a difference in the type inference implementation between .NET and Mono, the Mono build crashes with the following error:
Naiad/ConnectedComponents.cs(77,49): error CS0315: The type `Microsoft.Research.Naiad.Dataflow.IterationIn<TTime>' cannot be used as type parameter `TTime' in the generic type or method `Microsoft.Research.Naiad.Dataflow.StandardVertices.UnaryVertex<TInput,TOutput,TTime>'. There is no boxing conversion from `Microsoft.Research.Naiad.Dataflow.IterationIn<TTime>' to `Microsoft.Research.Naiad.Time<Microsoft.Research.Naiad.Dataflow.IterationIn<TTime>>'
The corresponding code looks like this:
public static Stream<Pair<TKey, TValue>, IterationIn<TTime>> StreamingAggregate<TKey, TValue, TTime>(this Stream<Pair<TKey, TValue>, IterationIn<TTime>> input, Func<TValu
e, TValue, TValue> aggregate)
where TTime : Time<TTime>
where TValue : IEquatable<TValue>
{
return Foundry.NewUnaryStage(input, (i, s) => new StreamingAggregateVertex<TKey, TValue, IterationIn<TTime>>(i, s, aggregate), x => x.First.GetHashCode(), x => x.First.GetHashCode(), "StreamingAggregate");
}Although this (somewhat exotic) type constraint is appropriately specific, it can be relaxed in the interests of compatibility (either to working on streams of any time type, or streams of the specific IterationIn<Epoch> time type).
When running on Mono with 3 or more processes, a Naiad computation will deadlock in the controller initialization phase.
To reproduce:
$ ./Examples.exe connectedcomponents 1000000 2000000 -t 1 -n 3 --local -p 0 &
$ ./Examples.exe connectedcomponents 1000000 2000000 -t 1 -n 3 --local -p 1 &
$ ./Examples.exe connectedcomponents 1000000 2000000 -t 1 -n 3 --local -p 2 &
The same program works for -n 2 -p {0, 1}. Thanks to the folks at ETH Zurich for reporting this issue.
Currently, the scheduler catches any exceptions in OnRecv or OnNotify callbacks, and invokes InternalComputation.Cancel(), which should terminate execution, by signaling the FrontierEmpty event. However, BaseComputation.Join() now blocks on a CountdownEvent that is signaled by each Scheduler on successful completion of Shutdown work, and this is not signaled in the exception case.
Hello Naiad developers,
We're using Naiad for a university project and we don't seem to be able to figure out how to pass non-primitive types between vertices. As advised in the readme we're using structs but it seems like messages are sent and never received.
We're still working with the release_0.4 branch since release_0.5 does not work with mono.
Here's the code to reproduce the issue https://gist.github.com/utaal/c10e450e1227f7ef5462 .
Please drop it in /Examples/Naiad, add examples.Add ("serialization", new Serialization.Serialization ()); to Program::Main. TestVertex2 is not receiving messages.
Is there any other known issue with the serialization framework? Are we using it incorrectly?
Thank you.
Hello Naiad Team!
I download the code and try to build the solution. But following errors occurs. Can you please give me some guidance how to configure the build setting? I follow the readme in the project but it doesn't work for me.
Severity Code Description Project File Line
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found Storage E:\info\naiad\Naiad-release_0.5\frameworks\Storage\CSC
Error CS1629 Unsafe code may not appear in iterators Naiad E:\info\naiad\Naiad-release_0.5\Naiad\CodeGeneration\AutoSerialization.cs 1242
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found Lindi E:\info\naiad\Naiad-release_0.5\Frameworks\Lindi\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found WorkGenerator E:\info\naiad\Naiad-release_0.5\Frameworks\WorkGenerator\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\Lindi\bin\Debug\Microsoft.Research.Naiad.Lindi.dll' could not be found Storage E:\info\naiad\Naiad-release_0.5\frameworks\Storage\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\WorkGenerator\bin\Debug\Microsoft.Research.Naiad.WorkGenerator.dll' could not be found Storage E:\info\naiad\Naiad-release_0.5\frameworks\Storage\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found AzureSupport E:\info\naiad\Naiad-release_0.5\Frameworks\AzureSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\Lindi\bin\Debug\Microsoft.Research.Naiad.Lindi.dll' could not be found AzureSupport E:\info\naiad\Naiad-release_0.5\Frameworks\AzureSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\frameworks\Storage\bin\Debug\Microsoft.Research.Naiad.Storage.dll' could not be found AzureSupport E:\info\naiad\Naiad-release_0.5\Frameworks\AzureSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found DifferentialDataflow E:\info\naiad\Naiad-release_0.5\Frameworks\DifferentialDataflow\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\Lindi\bin\Debug\Microsoft.Research.Naiad.Lindi.dll' could not be found DifferentialDataflow E:\info\naiad\Naiad-release_0.5\Frameworks\DifferentialDataflow\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found GraphLINQ E:\info\naiad\Naiad-release_0.5\Frameworks\GraphLINQ\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\Lindi\bin\Debug\Microsoft.Research.Naiad.Lindi.dll' could not be found GraphLINQ E:\info\naiad\Naiad-release_0.5\Frameworks\GraphLINQ\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\AzureSupport\bin\Debug\Microsoft.Research.Naiad.AzureSupport.dll' could not be found AzureExamples E:\info\naiad\Naiad-release_0.5\AzureExamples\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found AzureExamples E:\info\naiad\Naiad-release_0.5\AzureExamples\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\GraphLINQ\bin\Debug\Microsoft.Research.Naiad.GraphLINQ.dll' could not be found AzureExamples E:\info\naiad\Naiad-release_0.5\AzureExamples\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\Lindi\bin\Debug\Microsoft.Research.Naiad.Lindi.dll' could not be found AzureExamples E:\info\naiad\Naiad-release_0.5\AzureExamples\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\DifferentialDataflow\bin\Debug\Microsoft.Research.Naiad.DifferentialDataflow.dll' could not be found Examples E:\info\naiad\Naiad-release_0.5\Examples\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found Examples E:\info\naiad\Naiad-release_0.5\Examples\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\GraphLINQ\bin\Debug\Microsoft.Research.Naiad.GraphLINQ.dll' could not be found Examples E:\info\naiad\Naiad-release_0.5\Examples\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\Lindi\bin\Debug\Microsoft.Research.Naiad.Lindi.dll' could not be found Examples E:\info\naiad\Naiad-release_0.5\Examples\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found HdfsSupport E:\info\naiad\Naiad-release_0.5\frameworks\HdfsSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\Lindi\bin\Debug\Microsoft.Research.Naiad.Lindi.dll' could not be found HdfsSupport E:\info\naiad\Naiad-release_0.5\frameworks\HdfsSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\frameworks\Storage\bin\Debug\Microsoft.Research.Naiad.Storage.dll' could not be found HdfsSupport E:\info\naiad\Naiad-release_0.5\frameworks\HdfsSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\WorkGenerator\bin\Debug\Microsoft.Research.Naiad.WorkGenerator.dll' could not be found HdfsSupport E:\info\naiad\Naiad-release_0.5\frameworks\HdfsSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\AzureSupport\bin\Debug\Microsoft.Research.Naiad.AzureSupport.dll' could not be found NugetSample E:\info\naiad\Naiad-release_0.5\NugetSample\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\DifferentialDataflow\bin\Debug\Microsoft.Research.Naiad.DifferentialDataflow.dll' could not be found NugetSample E:\info\naiad\Naiad-release_0.5\NugetSample\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found NugetSample E:\info\naiad\Naiad-release_0.5\NugetSample\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Naiad\bin\Debug\Microsoft.Research.Naiad.dll' could not be found WebHdfsSupport E:\info\naiad\Naiad-release_0.5\Frameworks\WebHdfsSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\Lindi\bin\Debug\Microsoft.Research.Naiad.Lindi.dll' could not be found WebHdfsSupport E:\info\naiad\Naiad-release_0.5\Frameworks\WebHdfsSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\frameworks\Storage\bin\Debug\Microsoft.Research.Naiad.Storage.dll' could not be found WebHdfsSupport E:\info\naiad\Naiad-release_0.5\Frameworks\WebHdfsSupport\CSC
Error Metadata file 'E:\info\naiad\Naiad-release_0.5\Frameworks\WorkGenerator\bin\Debug\Microsoft.Research.Naiad.WorkGenerator.dll' could not be found WebHdfsSupport E:\info\naiad\Naiad-release_0.5\Frameworks\WebHdfsSupport\CSC
Hello,
We are running calculations on Naiad and computation fails on large datasets. For single thread and small datasets, there is no problem. We running Naiad on virtual machine in Mac.
Can you help to find out reason ?
Thank you.
Error :
Logging initialized to console
00:07:57.5913840, Graph 0 failed on scheduler 1 with exception:
System.NullReferenceException: Object reference not set to an instance of an obj
ect.
at Microsoft.Research.Naiad.Dataflow.VertexOutputBufferPerTime2.Send(TRecord record) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Frameworks\StandardVertic es.cs:line 313 at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.Conside rFlushingBufferedUpdates() in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime \Progress\UpdateAggregator.cs:line 157 at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.OnRecv( Dictionary2 deltas) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Progr
ess\UpdateAggregator.cs:line 77
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateProducer.Start() i
n c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Progress\UpdateProducer.cs:
line 92
at Microsoft.Research.Naiad.Scheduling.Scheduler.DrainMessagesForComputation(
Int32 computationIndex) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Sc
heduling\Scheduler.cs:line 343
00:07:57.5941910, Cancelling execution of graph 0, due to exception:
System.NullReferenceException: Object reference not set to an instance of an obj
ect.
at Microsoft.Research.Naiad.Dataflow.VertexOutputBufferPerTime2.Send(TRecord record) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Frameworks\StandardVertic es.cs:line 313 at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.Conside rFlushingBufferedUpdates() in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime \Progress\UpdateAggregator.cs:line 157 at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.OnRecv( Dictionary2 deltas) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Progr
ess\UpdateAggregator.cs:line 77
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateProducer.Start() i
n c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Progress\UpdateProducer.cs:
line 92
at Microsoft.Research.Naiad.Scheduling.Scheduler.DrainMessagesForComputation(
Int32 computationIndex) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Sc
heduling\Scheduler.cs:line 343
Logging initialized to console
00:07:57.5913840, Graph 0 failed on scheduler 1 with exception:
System.NullReferenceException: Object reference not set to an instance of an obj
ect.
at Microsoft.Research.Naiad.Dataflow.VertexOutputBufferPerTime2.Send(TRecord record) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Frameworks\StandardVertic es.cs:line 313 at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.Conside rFlushingBufferedUpdates() in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime \Progress\UpdateAggregator.cs:line 157 at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.OnRecv( Dictionary2 deltas) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Progr
ess\UpdateAggregator.cs:line 77
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateProducer.Start() i
n c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Progress\UpdateProducer.cs:
line 92
at Microsoft.Research.Naiad.Scheduling.Scheduler.DrainMessagesForComputation(
Int32 computationIndex) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Sc
heduling\Scheduler.cs:line 343
00:07:57.5941910, Cancelling execution of graph 0, due to exception:
System.NullReferenceException: Object reference not set to an instance of an obj
ect.
at Microsoft.Research.Naiad.Dataflow.VertexOutputBufferPerTime2.Send(TRecord record) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Frameworks\StandardVertic es.cs:line 313 at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.Conside rFlushingBufferedUpdates() in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime \Progress\UpdateAggregator.cs:line 157 at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.OnRecv( Dictionary2 deltas) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Progr
ess\UpdateAggregator.cs:line 77
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateProducer.Start() i
n c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Progress\UpdateProducer.cs:
line 92
at Microsoft.Research.Naiad.Scheduling.Scheduler.DrainMessagesForComputation(
Int32 computationIndex) in c:\Users\khgl\Desktop\codebase\Naiad\Naiad\Runtime\Sc
heduling\Scheduler.cs:line 343
The current inline serializer uses .NET serialization for strings, collections, tuples and other framework classes. (The default serializer specializes strings and other classes, but is less efficient for simple structs.) It would be worthwhile to specialize some common cases, in advance of deciding on a more pluggable serializer.
Hi Team Naiad,
I am fairly new to Naiad (and must say that it sounds really cool!). I was trying to write a simple Console Application in Visual Studio, where given a set of words, I can find the length of the smallest word. With each epoch of data, I want to calculate and return the length of the smallest word. To do so in Naiad,I wrote the following snippet:
var words = computation.NewInputCollection();
words.SelectMany(x => x.Split(' '))
.Min(w => w, w => w.Length )
.Subscribe(l => { foreach (var element in l) Console.WriteLine(element); });
With this snippet, I am not getting any word lengths at all, and I wanted to ask if I am using the functions right. Any advise/ help would be greatly appreciated!
Thanks in advance!
Best Regards,
Tanya
If a program decides to create a Naiad computation (either from Controller.NewComputation() or using a OneOffComputation), there is no way to cancel that computation, and no way to release the resources consumed by the underlying Controller.
Storage account keys should be fetched automatically for accounts attached to the user's default subscription. The submission client is running as the user, and can fetch them automatically. Getting them to the user executable running on the cluster is trickier. Perhaps the user should store desired storage account names in the app.config, and the submission client could populate them with keys as part of submission.
It seems that Naiad jobs run with only a single thread in Azure, even if a larger number of threads is requested.
I just want to run a pagerank program(Naiad/Examples/GraphLINQ/PageRank.cs) for graph analysis.When I run for a 1GB-Size graph,everything goes well; but when I run for a 3.6G-size graph(about 260,000,000 edges),the system prompted an exception followed:
Reading file C:\Naiad-release_0.5\rmat24.el
00:02:56.2408081, Graph 0 failed on scheduler 0 with exception:
System.OutOfMemoryException: Array dimension beyond the scope of support.
在 System.Collections.Generic.List1.set_Capacity(Int32 value) 在 System.Collections.Generic.List1.EnsureCapacity(Int32 min)
在 System.Collections.Generic.List1.Add(T item) 在 Microsoft.Research.Naiad.Frameworks.GraphLINQ.GraphCompactor1.OnReceive(Message2 message) 位置 C:\Naiad-release_0.5\Frameworks\GraphLINQ\GraphLINQ.cs:行号 1199 在 Microsoft.Research.Naiad.Dataflow.StandardVertices.UnaryVertex3.<>c.b__2_0(Message2 message, UnaryVertex3 vertex) 位置 C:\Naiad-release_0.5\Naiad\Frameworks\StandardVertices.cs:行号 446
在 Microsoft.Research.Naiad.Dataflow.Stage2.<>c__DisplayClass11_11.b__1(Message2 m) 位置 C:\Naiad-release_0.5\Naiad\Dataflow\Stage.cs:行号 383 在 Microsoft.Research.Naiad.Dataflow.ActionReceiver2.<>c__DisplayClass3_0.<.ctor>b__0(Message2 m, ReturnAddress u) 位置 C:\Naiad-release_0.5\Naiad\Dataflow\Endpoints.cs:行号 327 在 Microsoft.Research.Naiad.Dataflow.ActionReceiver2.OnReceive(Message`2 message, ReturnAddress from) 位置 C:\Naiad-release_0.5\Naiad\Dataflow\Endpoints.cs:行号 317
..........
how could I fixed this problem.I have tried to build the program on X86 and X64 platform,but could not fix the problem.
Thanks!
I am looking to do Tumbling Windows similar to what String Insight dose.
http://blogs.msdn.com/b/streaminsight/archive/2010/11/23/windows-part-1-the-basics.aspx
I do not currently see the operator in Naiad source.
Is it planned?
If not can you give guidance on how best to build it.
Just wondering why you have this rather than using Tuple?
There is a bug in LegacyLocalMailbox.Flush(RemoteAddress) that allows two worker threads to execute vertex code (in particular, the VertexInput.Flush() method). This causes unsafe access to the contents of the various message buffers in downstream mailboxes, which can lead to data corruption and crashes.
Hi again,
I've encountered another couple of issues when running https://gist.github.com/utaal/96c4328fbc48a07d69b4 on mono.
First, in distributed mode with two processes, process 1 does not seem to actually block on Sync. The following is the output of process 1.
Logging initialized to console
Pushing data
Syncing 0
00:00:02.7292913, Syncing at epoch (0) that is in the future of __Input0
Synced 0
Syncing 1
00:00:02.7400550, Syncing at epoch (1) that is in the future of __Input0
Synced 1
Syncing 2
00:00:02.7401329, Syncing at epoch (2) that is in the future of __Input0
Synced 2
Syncing 3
00:00:02.7401982, Syncing at epoch (3) that is in the future of __Input0
Synced 3
Syncing 4
00:00:02.7407855, Syncing at epoch (4) that is in the future of __Input0
Synced 4
Syncing 5
00:00:02.7408682, Syncing at epoch (5) that is in the future of __Input0
Synced 5
Syncing 6
00:00:02.7409424, Syncing at epoch (6) that is in the future of __Input0
Synced 6
Syncing 7
00:00:02.7410222, Syncing at epoch (7) that is in the future of __Input0
Synced 7
Syncing 8
00:00:02.7410966, Syncing at epoch (8) that is in the future of __Input0
Synced 8
Syncing 9
00:00:02.7411738, Syncing at epoch (9) that is in the future of __Input0
Synced 9
(...)
I was watching the output of process 0 and by this time the subscription did not even finish outputting data for epoch 1.
Is this behavior somehow expected? Do you believe it may be mono-specific?
Moreover, occasionally, when running with -t > 1 we either get the following exception mid-computation - all the processes stop working immediately after this - or the computation just gets stuck.
00:00:03.2077718, Graph 0 failed on scheduler 0 with exception:
System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at System.Collections.Generic.Dictionary`2[Microsoft.Research.Naiad.Runtime.Progress.Pointstamp,System.Int64].get_Item (Pointstamp key) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.ConsiderFlushingBufferedUpdates () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.OnRecv (System.Collections.Generic.Dictionary`2 deltas) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateProducer.Start () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Scheduling.Scheduler.RunWorkItem (Int32 graphId) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Scheduling.Scheduler.InternalStart () [0x00000] in <filename unknown>:0
00:00:03.2084448, Cancelling execution of graph 0, due to exception:
System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at System.Collections.Generic.Dictionary`2[Microsoft.Research.Naiad.Runtime.Progress.Pointstamp,System.Int64].get_Item (Pointstamp key) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.ConsiderFlushingBufferedUpdates () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateAggregator.OnRecv (System.Collections.Generic.Dictionary`2 deltas) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Runtime.Progress.ProgressUpdateProducer.Start () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Scheduling.Scheduler.RunWorkItem (Int32 graphId) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Scheduling.Scheduler.InternalStart () [0x00000] in <filename unknown>:0
I apologize for the missing file names / line numbers, I was not able to reproduce the issue when running with --debug.
Thank you.
Andrea
It appears that the default behavior of Mono's CodeDOM is to treat warnings as errors, even when the TreatWarningsAsErrors option is set to false. Running Naiad in distributed mode yields the following type of error:
Fatal: errors occurred during the generation of a serializer for type Microsoft.Research.Naiad.Serialization.MessageHeader:
/tmp/4jebzyce.gpz/ReleaseObjects/5b99a60e.0.cs(70,11) : warning CS0219: The variable `bytesRemaining' is assigned but its value is never used
/tmp/4jebzyce.gpz/ReleaseObjects/5b99a60e.0.cs(145,11) : warning CS0219: The variable `bytesRemaining' is assigned but its value is never used
/tmp/4jebzyce.gpz/ReleaseObjects/5b99a60e.0.cs(200,11) : warning CS0219: The variable `bytesRemaining' is assigned but its value is never used
Unhandled Exception:
System.Exception: Code generation for Microsoft.Research.Naiad.Serialization.MessageHeader failed
at Microsoft.Research.Naiad.Serialization.AutoSerialization+SerializationCodeGeneratorForType.GenerateSerializer[MessageHeader] () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Serialization.BaseSerializationCodeGenerator.GetSerializer[MessageHeader] () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Runtime.Networking.TcpNetworkChannel.get_HeaderSerializer () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Runtime.Networking.TcpNetworkChannel.AnnounceStartup (Int32 barrierId) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Runtime.Networking.TcpNetworkChannel.DoStartupBarrier () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.BaseController.DoStartupBarrier () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.BaseComputation.Activate () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.InternalOneOffComputation.Activate () [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Examples.Throughput.Throughput.Execute (System.String[] args) [0x00000] in <filename unknown>:0
at Microsoft.Research.Naiad.Examples.Program.Main (System.String[] args) [0x00000] in <filename unknown>:0
The culprit is a stray declaration of a "bytesRemaining" variable in the TrySerializeMany method of each generated serializer.
Currently, these projects carry a dependency on the WindowsAzure.Storage NuGet package, which is not fetched automatically by the build process.
Hi Naiad developers, the iterative model in Naiad is quite attractive. But I am wondering what if the previous round of computation keeps going but the new event comes in? For example, in connected component case, let's say edge <i, j> comes in and triggers some computation. Before the computation ends, edge <m, n> arrives. So will the edge <m, n> be buffered? Or the two rounds of computations can be performed simultaneously? Thanks!
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.