TPL Dataflow - An [Extremely Short] Introduction

In the process of preparing my talk for the 2012 Fox Valley Day of .NET, I decided that the best way to get a reasonable talk laid out is to write a blog series about it

.  Maybe that means I’m weird, I don’t know, but it does mean that I’ll actually have some new entries for the blog after far, far too long.  I need to strike that balance between ‘long ass boring’ and ‘you are done already?’, hopefully this will be a good way to get there.  I’ll go into a lot more detail about the project and resulting solution here than I ever could in a talk, so I’ll apologize in advance if you’re attending and see something here you were hoping I’d elaborate on in my session.  If you see these, and have specific questions, I’ll be glad to answer them here or in person.

So, obviously, I’m not going to outline the history of multi-threaded programming, or even the complete history of multi-threaded programming in the context of the .NET Framework. I will start out trying to set the context for the world we’re currently faced with, and lay out what I hope to accomplish in this series. Future articles will delve more deeply into specific topics around Dataflow, and we’ll see our simple app hopefully mature into a full-fledged test-bed for Dataflow experimentation.

 

Where are we?

The 4.0 .NET Framework introduced a new concept for dealing with multi-threaded programming, the Task Parallel Library [or TPL from here on out].  The TPL sought to pull out the ‘good’ bits of parallel programming, from both best practices to burnt fingers, and roll them into a nice, easy-to-use framework.  For the most part, at least in my experience, Microsoft succeeded in this goal.  Certainly it wasn’t perfect, and there were bits missing [many of which appear in the .NET 4.5 preview], but it was an excellent ‘1.0’.

Those who used the TPL built up various solutions for retrieving the data to process, bundling it up, spinning up tasks to do the processing and generally ‘managing’ the flow of data through your application to get your job done.  Stephen Toub explained this approach very well in his 2011 //BUILD session, talking about the ‘data first’ nature of the existing TPL and some of the limitations of the approach.  TPL Dataflow [TDF] really turns things on their head a bit, switching to a ‘computation first’ model, whereby you setup the computation and then ‘wait’ for data to flow into the network for processing.

To be sure, everything you can do with TDF you can do with TPL or TPL+Rx or TPL+(your library of choice), and many of you have already built similar solutions to the framework.  What’s exciting about TDF is that a lot of the ‘stuff’ that needed to be managed as part of the ‘overhead’ of a project is now handled ‘out of the box’, and we can move from concept to solution quickly and spend more time on the rules/scalability/robustness/documentation side of the equation than we ever had time for previously.

 

What is TPL Dataflow [TDF] Anyway [in 100 words or less]?

TPL Dataflow is a set of Interfaces (IDataflowBlock, ITargetBlock, ISourceBlock and IPropagateBlock) and some implementing classes that can be used to build data processing networks.  The classes are located in the System.Threading.Tasks.Dataflow namespace, and are part of .NET Framework 4.5 [currently in beta].  The interfaces/classes are designed from the ground up to support multi-threaded, asynchronous programming and can be used to solve many classes of problems that were difficult to do with the TPL natively.

Sidebar: These are .NET 4.5 classes, so you WILL NOT be using them in new ‘Windows 8/RT Metro’ applications as they are not part of the surface area exposed by the new Windows Runtime.  You may be able to use them ‘wrapped’ in a WinRT component, but I haven’t explored that option yet, and I’m not sure that you’d want to in any case.  You WILL be using these classes on the ‘backend’ that you write to service all these ‘consumption’ Metro apps that you are building.

 

So Where am I Going With This?

The long term goal is a series of articles featuring these little guys (OK, so yes, they are currently big):

I’m sure you’re as pleased to meet them as they are to meet you.

  I hope to use these TPLemmings to help demonstrate how to use the various classes that TDF offers, and have a little fun at the same time!  Unfortunately you won’t be able to do any more than meet them today, but I do hope you’ll come back and say ‘Hi’.  After all they’ve got stuff to show you!

 

Conclusion

This article really serves as an introduction to the series, and gives you an idea of what TPL Dataflow is.  Future articles will cover the interfaces and classes supplied by TDF and go into detail on what types of problems they can solve.  And because I’d be remiss if I didn’t include some ‘code’ in the article, I’ll close with this little snippet.

Save the image above to your HDD, and then create a new [empty] WPF application.  Include the PNG file in an ‘Assets’ folder in the project, and then paste the following XAML into MainWindow.xaml:

    <Image x:Name="lemming" Source="Assets/TPLemming.png" Stretch="None" HorizontalAlignment="Center" RenderTransformOrigin="0.5,0.5">
        <Image.Resources>
        <Storyboard x:Key="Walk" AutoReverse="True" RepeatBehavior="Forever">
                <DoubleAnimationUsingKeyFrames Storyboard.TargetProperty="(UIElement.RenderTransform).(TranslateTransform.X)" Storyboard.TargetName="lemming">
                    <DiscreteDoubleKeyFrame KeyTime="0:0:0.5" Value="-55" />
                    <DiscreteDoubleKeyFrame KeyTime="0:0:1" Value="61" />
                </DoubleAnimationUsingKeyFrames>
                <PointAnimationUsingKeyFrames Storyboard.TargetProperty="(UIElement.Clip).(PathGeometry.Figures)[0].(PathFigure.StartPoint)" Storyboard.TargetName="lemming">
                <PointKeyFrameCollection>
                    <DiscretePointKeyFrame KeyTime="0:0:0.5" Value="122,0" />
                    <DiscretePointKeyFrame KeyTime="0:0:1" Value="0,0" />
                </PointKeyFrameCollection>
            </PointAnimationUsingKeyFrames>
            <PointAnimationUsingKeyFrames Storyboard.TargetProperty="(UIElement.Clip).(PathGeometry.Figures)[0].(PathFigure.Segments)[0].(LineSegment.Point)" Storyboard.TargetName="lemming">
                <PointKeyFrameCollection>
                    <DiscretePointKeyFrame KeyTime="0:0:0.5" Value="244,0" />
                    <DiscretePointKeyFrame KeyTime="0:0:1" Value="122,0" />
                </PointKeyFrameCollection>
            </PointAnimationUsingKeyFrames>
            <PointAnimationUsingKeyFrames Storyboard.TargetProperty="(UIElement.Clip).(PathGeometry.Figures)[0].(PathFigure.Segments)[1].(LineSegment.Point)" Storyboard.TargetName="lemming">
                <PointKeyFrameCollection>
                    <DiscretePointKeyFrame KeyTime="0:0:0.5" Value="244,244" />
                    <DiscretePointKeyFrame KeyTime="0:0:1" Value="122,244" />
                </PointKeyFrameCollection>
            </PointAnimationUsingKeyFrames>
            <PointAnimationUsingKeyFrames Storyboard.TargetProperty="(UIElement.Clip).(PathGeometry.Figures)[0].(PathFigure.Segments)[2].(LineSegment.Point)" Storyboard.TargetName="lemming">
                <PointKeyFrameCollection>
                    <DiscretePointKeyFrame KeyTime="0:0:0.5" Value="122,244" />
                    <DiscretePointKeyFrame KeyTime="0:0:1" Value="0,244" />
                </PointKeyFrameCollection>
            </PointAnimationUsingKeyFrames>
        </Storyboard>
        </Image.Resources>
        <Image.Triggers>
            <EventTrigger RoutedEvent="FrameworkElement.Loaded">
                <BeginStoryboard Storyboard="{StaticResource Walk}" />
            </EventTrigger>
        </Image.Triggers>

        <Image.RenderTransform>
             <TranslateTransform X="61"/>
        </Image.RenderTransform>
        <Image.Clip>
            <PathGeometry>
                <PathFigure IsClosed="True" StartPoint="0,0">
                    <LineSegment Point="122,0" />
                    <LineSegment Point="122,244" />
                    <LineSegment Point="0,244" />
                </PathFigure>
            </PathGeometry>
        </Image.Clip>
    </Image>

Enjoy!