tag:blogger.com,1999:blog-35945392024-02-07T21:16:39.179-08:00Muse of the Mind...Some basics and the maybe more on things I know, research and pass on to others.. Potentially random, but with meaning...Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.comBlogger41125tag:blogger.com,1999:blog-3594539.post-90331487561740676432009-12-16T01:06:00.001-08:002009-12-16T01:06:44.360-08:00Moving blog and its contents to Wordpress over the next week.Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-85762728013296317332009-12-09T21:01:00.001-08:002009-12-15T02:49:26.541-08:00As the title states, I am now delving into Google's NOSQL BIG TABLE architecture to see how it is looking as part of my research into cloud based data integration for very large/high transaction volume systems.<br />
Maybe a few blog posts will entail?Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-38637693796653839682009-12-09T20:58:00.001-08:002009-12-09T21:28:41.815-08:00DB Server Latency can kill data intensive processing systems..<p>Sounds obvious, yet many forget this… </p> <p>Take an example that is pretty recent, a prototype spike of functionality was created that for a certain reason had to read a lot of data from many sources, and then write aggregated results a row and sometimes a column at a time. Now there could be millions of pieces of data being written real-time.</p> <p>The issue here was how to increase performance, especially when doing many thousands of writes to a single database server at a time. In-fact we hit 400ms latency issues with SQL server running out of ports and being able to process requests and connections. </p> <p>Of course we got performance increase buy running multiple threads of updates at once, but was still at the mercy of the single update/command latency.</p> <p>Another way around this type of processing was to do all work as batch/set based updates, but there are issues in auditing data.</p> <p>The best was so far was to create functions inside of SQL Server as native .NET but was platform fixed but also was not as useful as expected.</p> <p>The above scenario and the many other times I have run into it, is one of the reasons I am looking at Cloud/parallel data systems and also BIG TABLE or NoSQL based data management systems both in memory and on disk.</p> <p>The landscape is getting interesting, but as always in most cases teh performance we got with SQL is more than enough, however I wanted to see just how much more performance we can get. Especially when looking at very large data integration/merge/match projects or data management and data updates from many 100’s,1000’s, 100000’s of potential systems or users at once.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-21943018473657924252009-12-09T20:29:00.001-08:002009-12-09T20:29:29.755-08:00Infrastructure Scaling-up Vs. Scaling-Out.<p>It is quite common in web applications or distributed hosted systems to look at issues in scaling your platform to either support more clients or processing speed/power.</p> <p>There are 2 approaches to upgrading your hardware.</p> <ul> <li><strong>Scaling-up</strong>  - the process where a single ‘Iron Server’ is upgraded to a maximum load. This means you upgrade a server to support the maximum CPU/RAM allowed. On Some bare bone ‘Iron Servers’ these statistics can be quite impressive for example a <strong>HP Proliant DL785 G5</strong> can be upgrade to 8 CPU sockets, 64 memory sockets, 16 drive bays, 11 potential expansion slots and 6 power supplies to provide the server power.</li> <li><strong>Scaling-out – </strong>This is where a moderate powered server based setup is augmented by adding more equivalent powered servers Ideally with exactly the same architecture or specification although technology advance make this hard over time.</li> </ul> <p>With the advent of cloud computing hitting a more mainstream market and some proven viability. It is actually quite natural to look at scaling-out a system to take advantage of this newer parallel processing paradigm.</p> <p>Now there are issues you need to be aware of in both scenario’s so lets start with the obvious issues and yes all are mainly TCO/ROI based points for now. Later I will do some posts on the development and systems impact of these ways of achieving physical scalability. ( I am listing negatives only ).</p> <p><strong>Scaling-UP</strong></p> <ul> <li>Cost can be quite prohibitive for a single server albeit a monster as the HP Proliant can be when fully loaded.  (I have seen some priced at over 125K+ GBP (200K+ USD)</li> <li>Potentially a single point of failure</li> <li>Not as optimized for parallel processing as am array/gird of servers and does not scale the same</li> <li>Physical limit on scaling (cpu/ram/storage) storage can be made external so is not an issue.</li> </ul> <p><strong>Scaling-Out</strong></p> <ul> <li>Rack Space/Server space costs more</li> <li>Power consumption can be higher for same amount of processor/memory when compared to a scaled-up iron server</li> <li>OS and Database Licensing costs can be a lot more expensive, however this is changing with cloud database and application servers.</li> <li>Infrastructure to support and manage the servers can cost a lot in both time and money.</li> </ul> <p>Final notes, if the real issue is cost; Although scaling-up seems more expensive do not underestimate the sheer amount of resource costs in supporting many servers, storage of the servers, network infrastructure and so on. The only time scaling out seems to make a real case on cost is when you have a critical mass in scaling to meet client needs, and even then you can cloud/cluster a collection of “iron servers” </p> <p>Personally, it depends on the situation you are faced with, if cost is an issue but you have some good network infrastructure clouding with a array of servers is a very viable options. But one you start to scale it maybe worthwhile just getting an iron server or 2 and scaling them up in the same network. – Scaling out is really useful when you have a lot of old servers/pc’s that are already inside your infrastructure and can be utilized even if just a little by a cloud or grid based solution.</p> <p>I have purposely kept aware from the area of server virtualization and provisioning as it can change things drastically. This is also why I have avoided the issue of OS choice as well.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-51446682070826930582009-12-09T19:59:00.001-08:002009-12-09T20:03:37.289-08:00high performance high user websites.<p>While looking at performance in eCommerce and large social sites on the web I come across a couple surprising profiling reports. The most impressive to me so far at least is that of Plenty of Fish, a very popular free online web dating site that has been priced up at near 1billion USD. this itself id not that surprising in the big stakes of online systems. What is surprising is the amount off hits and users their site gets and more importantly just how little hardware and how simple an architecture their system is. It’s truly beautiful in this day and age of bloat and high expense to implement server arrays and grids.</p> <p>Here’s some basic stats:</p> <p>Plenty of fish gets approximately 1.2billon pages/views a month with an average 500,000 unique logins per single day. That's not bad at all. </p> <p>the System deals out 30+million hits a day, around 500-600 hits a second. Now that is very impressive!</p> <p>The technology used is ASP.NET at its core,.The server setup can apparently deal with 2 million page views a hour under stress</p> <p>Now the hardware!!! and this is quite impressive:</p> <p>2 load balanced web servers (to get past the IIS restriction of 64,000 simultaneous connections at once issue.</p> <p>3 Database servers, a main account server and 2 others used for searching and at a guess profiling of data. ( I do suspect these are seriously overloaded ‘Iron Servers’ which in some cases can cost easily anywhere between 50-100K GBP)</p> <p>Despite all of this, it’s pretty impressive a site can reach that levl of performance using so little hardware.</p> <p>Over the next week or two I will explore similar architectures and also the additional application of cloud/parallel processing across a Service buss to see if anything can get close to those stats (although I don’t have that hardware, some things can be approximated). I do think a virtual cloud running on  suitable batch of 5 decent machines should be able to deal with 200-500K web serves an hour with a moderately complex media rich (images not video) dynamically generated system. I will also be looking at issues of transactions over boundaries as this is where I see the biggest hit on performance. (database clusters/clouds accessed by cloud or service applications for example .NET applications or java clients if REST is used)</p> <p>I will also look at the issue of scaling out Vs Scaling up the physical architecture.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-583817872652002552009-12-09T14:41:00.001-08:002009-12-09T14:41:28.919-08:00Just what is a complex technology?Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-7785038098462173822009-12-08T14:26:00.001-08:002009-12-08T14:26:48.785-08:00DSL memory management and parameter issues…<p>Ok now, I am doing a little work on the DSL for this blog. I am currently working on the Runtime stack and Activation record support for interpreter memory management when interpreting and executing DSL based code/logic.</p> <p>Before I go into specifics, I will outline some key concepts/terminology that will make this a lot easier to explain.</p> <ul> <li>Memory Maps – A memory map contains a collection of memory cells. A memory map is pretty much a key part of an activation record. For now assume a memory map is a collection of variable/parameter data in an activation record for a given function or scope of processing.</li> <li>Memory Cells – is where actual data is stored, and is part of a memory map. Here is where the actually details of a variable/parameter is stored and its current value.</li> <li>Activation Records – An Activation record exists for each level of scope in the interpreter as it runs code. There is an activation record for each level of a interpreted function call. Even if a recursive function is called there will be a separate activation record for each call. You can also access any activation record above the one you are currently in to access scope wide variables in a chained function call.</li> </ul> <p>Ok now some nuts and bolts, consider the following piece of pseudo code:</p> <p><strong>START OF PROGRAM</strong></p> <p>     LOCAL VARIABLE StringX EQUALS “String X”</p> <p>     LOCAL VARIABLE StringY EQUALS 2010</p> <p>     PRINT RESULT OF</p> <p>           FUNCTION COMBINE USING StringX ADN StringY</p> <p>END OF PROGRAM</p> <p> </p> <p><strong>DEFINE FUNCTION COMBINE (X, Y)</strong></p> <p>      RETRUN StringX+StringY</p> <p>END OF FUNCTION</p> <p> </p> <p>The picture below shows a diagram of a memory map and the cells inside of it.</p> <p> <img style="border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px" title="image" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfzrZ6LUbBJgd5bM-IKcyizgGeZ9AVl9ATGfbsJn6JAnaj7cXf3sQYbg_bWYfH4Ne5RmzKRuh9FrT4GJOODhA6Q8k6ZHEr5Y4dauBLbqB8ekf0pA1HBLQv5EFZRyQnBCCzbRpZPg/?imgmax=800" width="334" height="127" /> </p> <p>As you can see the top Level of this memory map, shows that 2 variables StringX and StringY where created. Not only that when the function is called, both StringX and StringY are copied into memory map 2 variables X and Y when they are passed as parameters to the COMBINE function. This is an example of pass by value parameter passing.</p> <p>Ok now that is covered look at the diagram illustrates an example runtime stack with Activation records as it goes through the previous pseudo code.</p> <p><img style="border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px" title="image" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFUuPeIBS1z9EuTP2a2jqo_QFMxhAj4_1vAQcvDyngJPYZUY74VT85UK9ooUTff1eNB3QXtC5iOKbnjutckvhgmiLk-EoW8C4CKgdxhUj5_ca5fIhndQZlemeZLjkAg9YZFBIntA/?imgmax=800" width="448" height="243" /> </p> <p>Now from this example you can see the transient states of the runtime stack as as it runs through the code.</p> <p>First we just have variables defined in the Main function under AR1 (Active record 1). As the function combine is called, the AR1 active record creates an entry for the result variable (the result of a function call in this pseudo language). and then processing passes to the function COMBINE; which itself creates a couple entries in a new active record AR2. (which you can see is pushed onto the active record stack in the runtime memory system.)</p> <p>Once processing in the function COMBINE is complete processing is handed back to the calling MAIN program and the Active record AR2 is popped from the stack.</p> <p>A key thing to remember here is that all Active records can see any data in lower level Active records in the Runtime stack. This becomes very important later when I will explain how to define varying levels scope to a variable in a DSL. </p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-34064636255634146642009-12-08T12:53:00.001-08:002009-12-08T12:53:19.822-08:00Basic High level design of a text based matrix parser…<p>In this post I will introduce the basic architecture for a matrix based text file/stream parser that can deal with file formats in both fixed width notation and delimited formats. Not only that the matrix parser can deal with 1-2-1, 1-2-M and M-2-1 relations ships in the text stream. It can also deal with mixed format files where you have column based entries, CSV formats and also multiple ragged formats of fixed width data.</p> <p>The concept is in reality quite simple, and if you think about it after looking at the architecture diagram below. Is quite similar to the structure used when creating complex reports in tools liek crystal reports and so on.</p> <p>The structure is quite simple, here is a text breakdown of the key parts of a matrix parser.</p> <ul> <li>Scanner</li> <li>Token Extractor</li> <li>Matrix structure made up of</li> <ul> <li>Blocks of rows that can be CSV, fixed, column meta-type and made up of</li> <ul> <li>rows which can contain tokens or made up of </li> <ul> <li>columns which can contain</li> <ul> <li>another Block or the data to parse</li> </ul> </ul> </ul> </ul> </ul> <p>Below is a high level class diagram for a simplistic Matrix based text parsing library/API. ( the scanning/parsing/tokenizing part is taken out.</p> <p> </p> <p><img style="border-bottom: 0px; border-left: 0px; display: inline; border-top: 0px; border-right: 0px" title="image" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFr64kg4JBbwIU1D17E9WJ5wlJg6ahfhz8MEXy4iAt1K9g6IGDuToBN6hCEqn_HN2sbtNKCAzdzpxa4lWReB_rlyZo0I49eInDwTBWuKU_uKnpRYy-D3a5wRSAK6MbgCN1tQAEKg/?imgmax=800" width="464" height="360" /> </p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:8eb9d37f-1541-4f29-b6f4-1eea890d4876:a20b9c2a-e127-40ee-a26d-2d99468c15f9" class="wlWriterEditableSmartContent"><p><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJ5OfFHHi25YDK9UY-jyTZLavphjYiP8KmWmDhd7ERLARJZQJhxR0Fd1PSSMUkO3vlXtUxd8h9LYB0ZUR6zZajdbYFCDg52xA2V6DwgSsatnkw3CQhPT5S5aDpKL3QBB9ZIbxyrQ/?imgmax=800" target="_self">MatrixTextParser-HL.png</a></div></p></div> <p>I will put up a full architecture later, when I walk through how the scanner/parser and token extractor works.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-73122028359886027602009-12-08T11:17:00.001-08:002009-12-08T11:19:04.027-08:00DSL and logic/rule systems…<p>Ok another post on logic/rule systems and their implementation through a DSL (Domain Specific Language).</p> <p>Most rule engines use very simplistic logic for making choices based upon conditions, these systems can easily be created as a DSL with basic IF/THEN/ELSE constructs and then functional processing</p> <p>These logic systems I refer to as simple procedural logic based systems. </p> <p>Another area of definition in DSL/Logic/Rule engines is whether the engine itself is to do any processing, or just simple returns the result of a condition as a logical Boolean value or a piece of data. (This would be a very simple DSL, very limited grammar and little else except for an execution based interpreter.)</p> <p>There are of course many Pro’s and Cons to both ways of doing things, and then there is the hybrid where both techniques are implemented. (this is a simple interpreter focused DSL, with limited grammar and context management and symbol table)</p> <p>The next level of complexity in a logic system is where you extend the basic syntax to take into account far more complex processing and logical conditions or indeed logic chaining. At essence a lot of these logic/rules engines actually have a full scripting language which is focused on the domain of the logic being looked at. </p> <p>This is obviously a very clear case for a fully realized DSL with full interpretation support, and if performance is a real issue the DSL should also compile to the platform it is running on. (in .NET that would be to MSIL in the CLR, in the Java World it is to a JVM realized format.)</p> <p>Finally, there are also 2 ways to implement a DSL besides the pure language syntax approach these are in order of my personal preference.</p> <ul> <li><strong>Template focused against DSL</strong> – this is where you have a logic template and you map values into the template prior to processing, since you are not running all script through a DSL unless you are working with expressions. It has a significant speed increase. If Architected well you outperform DSL interpretation. The reason I prefer this format is that you can also mix in DSL based script at any point, and thus provide incredible flexibility. You can also implement focused parsing techniques to only interpret/error check/process logic for a specific structure of code for a domain. Which also allows a lot of very specific processing to be done within a known sandbox and or context of a system. </li> <li><strong>Template boxed no DSL</strong>  - Otherwise known as lazy DSL, this way of doing things simple means you have an execution layer that has execution profiles like an interpreter but can only do simple logic based on a template, these systems start of very well. However once users start to create their own logic/rules in most cases users will demand more and more control and processing flexibility which necessitates basic DSL and Expression parsing execution. </li> </ul> <p>An example of a boxed template for the <strong>IF <condition> THEN <process true result> ELSE  <Process false result></strong></p> <p>to execute the above you would just have a simple function</p> <p><strong>Bool ExecuteBoxedIFTHENELSE( conditionExpr, trueProcess, falseProcess) –> Returns logical true/false</strong></p> <p><strong>{</strong></p> <p><strong>if ( ExecuteExpression( CondExpr) ) </strong></p> <p><strong>{ExecuteProcess(trueProcess); return true; }</strong></p> <p><strong>else </strong></p> <p><strong>{ ExecuteProcess(falseProcess); return false;)</strong></p> <p><strong>return false;</strong></p> <p><strong>}</strong></p> <p>pretty simple, and if you create the functions Execute condition to evaluate and only return a true or false value; you can see how easy it is to either nest this logic and evaluate it.</p> <p>But the key thing to think about is whether you need to do advanced processing using variables, memory management, context and chaining of rules. It is also very hard to debug this format of rule at runtime inside the system that implemented it, since there is no scanning and interpreting in a traditional manner, thus syntax checking and lookup in a rule builder UI is a challenge to say the least. however for simple rules and logic it is very fast to implement and understand conceptually, whereas writing a DSL is a far more complex task.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-85323628343712589572009-12-04T12:32:00.001-08:002009-12-04T12:32:47.337-08:00Thick Vs. Thin Client Vs Rich Client.<p>Yet another argument I hear all of the time, crazy really as all clients have their place. However before  I put my completely biased view forward lets dispel some far too common myths.</p> <p>For now this post will focus purely on non-mobile device based (User-Interfaces) and UX. (User-Experiences).</p> <p><strong>Thick Client Myths</strong></p> <ul> <li><strong>Better UI</strong> – well yes and no it really depends on the tools you use and he skills of your team. I have seen some truely atrocious User Interfaces in my career and the majority of them have been windows/linux thick client based. </li> <li><strong>Best User experience</strong> – pretty much the same as above, think clients do excel in UI performance providing the workstation it is installed on is powerful enough. </li> <li><strong>Weak in distributed computing</strong> – I love this one, it does nto come up often but it seems that in all of the PR that web services go in early 2000 actually got people thinking you can only really use distributed data in rich(smart) or web based client UI’s. </li> <li><strong>Hard to deploy around a company</strong> – My personal favorite, for a long time now there has been a big shift in deployment options across an enterprise (or SMB) where you can have applications auto-install across a network based on user privileges. Not only that applications can also be self updating once installed. This actually get’s around the deployment issue normally associated with think clients. OF course a web browser application is immediately available; However, you must also realize that a web application has a lot of restrictions and also has some pretty big security and deployment issues also. </li> </ul> <p><strong>Web Client Myths</strong></p> <ul> <li><strong>.Easy to Deploy</strong> – This depends entirely upon yor organization and the technologies used on a website. A very good example of this is flash and java enabled applications which need a browsers security to be reduced so these technologies can be run effectively. </li> <li><strong>Bad UI features</strong> – Where to start, some of the best UI I have seen have been web based. However as technology and user expectation grow. I see more and more web based applications trying to do very complex processing and logical displays in a web front end that really push the web to it’s limits and ultimately fails to deliver a good user experience. Examples of these are UI’s that are screen designers/workflow layout designers. Most of these web applications resort to very large Java code/applications to be installed locally. Or large JavaScript/VBScript files that get processed on every web page and webpage refresh that just means post backs or Async callbacks are slow at best. Also there is the unwritten rule in web UI design that ideally you never go beyond 50-75 UI controls on a single page. </li> <li><strong>Good User Experience</strong> –This really is a graphic and UX/UI design area. If you have a great design team that understands issues of the backend providing data, and also the REAL practical limitations of a web application. The UX should be good. However, most web applications (not websites) are pretty weak user experiences outside of the pure media and gaming industries. </li> <li><strong>Cheap to Implement</strong> – This is the biggest pain in my Ass myth. I have never been on web application project that has been cheaper to implement than a Thick client, mobile, rich client application. Most of the time is usually spent trying to get the UI to work well in more than one browser and also to maintain state in a complex page. I have seen systems with workflow designers taking months to complete in a acceptable web experience compared to a think client, again mostly due to state and application control issues. For simple UI/UX the web is amazing fast to prototype and get running, for commercial quality user experience and UI flexibility the web is not the way to go. </li> </ul> <p><strong>Rich Client Myths</strong></p> <ul> <li><strong>Security – </strong>The number one reason for not using a RIA is security, because an application can be downloaded from the web, uses 80% web technology, but seems to run as a standalone thick client like experience on a desktop or in a web browser concerns people. This also leads onto the next myth. </li> <li><strong>Deployment – </strong>The deployment issue is tied more to security and corporate/company standards for staff workstation more than it is on actual data security issue. The issue here is that a lot of companies do not allow staff to download and install browser plug-in; quite a few even disable flash players. All RIA technologies currently need a client installing in a browser or on the operating system in use. Example RIA frameworks are </li> </ul> <p>So when would I use each of these types of UI, well that will be in a later post, my lunchtime just ran out…</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-19400691037514922182009-12-04T00:10:00.001-08:002009-12-04T00:50:12.281-08:00Implementing the Levenshtein distance algorithm<p>In a previous post I mentioned there are many ways to perform a test on a string to see how closely it can match another string. One of the more interesting algorithms for this is the <strong>Levenshtein Distance Algorithm</strong>.</p> <p>This algorithm is named after Vladimir Levenshtein in 1965. It is often used in applications that need to determine how similar or indeed how different two string values are.</p> <p>As you can guess this is very useful if you every want to implement a basic fuzzy matching algorithm for example. It is also used a lot in spell checking applications to suggest potential word matches.</p> <p>The Code sample below is in C# but can be translated to any language. It is not optimized at all and is only here to serve as an illustration. It should be noted that there are better and more sophisticated Distance algorithms that you can find via Google or a good book on string comparison algorithms. (there are a lot of ways to re-factor and optimize the following code.</p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:9ce6104f-a9aa-4a17-a79f-3a39532ebf7c:18c0190e-a140-4d4a-9366-28b4108e3718" class="wlWriterEditableSmartContent"> <div style="border: #000080 1px solid; color: #000; font-family: 'Courier New', Courier, Monospace; font-size: 10pt"> <div style="background-color: #ffffff; max-height: 300px; overflow: auto; padding: 2px 5px; white-space: nowrap"><span style="color:#0000ff">class</span> <span style="color:#2b91af">LevenshteinDistance</span><br> {<br> <br>      <span style="color:#0000ff">public</span> <span style="color:#0000ff">int</span> Execute(<span style="color:#2b91af">String</span> valueOne, <span style="color:#2b91af">String</span> valueTwo)<br>     {<br>         <span style="color:#0000ff">int</span> valueOneLength = valueOne.Length;<br>         <span style="color:#0000ff">int</span> valueTwoLength = valueTwo.Length;<br> <br>         <span style="color:#0000ff">int</span>[,] DistanceMatrix = <span style="color:#0000ff">new</span> <span style="color:#0000ff">int</span>[valueOneLength, valueTwoLength];<br> <br>         <span style="color:#0000ff">for</span>(<span style="color:#0000ff">int</span> i=<span style="color:#a52a2a">0</span>; i<valueOneLength; i++)<br>              DistanceMatrix[i, <span style="color:#a52a2a">0</span>] = i;<br>        <br>         <span style="color:#0000ff">for</span>(<span style="color:#0000ff">int</span> j=<span style="color:#a52a2a">0</span>; j<valueTwoLength; j++)<br>             DistanceMatrix[<span style="color:#a52a2a">0</span>, j] = j;<br> <br>         <span style="color:#0000ff">for</span>(<span style="color:#0000ff">int</span> j=<span style="color:#a52a2a">1</span>;j<valueTwoLength;j++)<br>             <span style="color:#0000ff">for</span> (<span style="color:#0000ff">int</span> i = <span style="color:#a52a2a">1</span>; i < valueOneLength; i++)<br>             {<br>                 <span style="color:#0000ff">if</span> (valueOne[i] == valueTwo[j])<br>                     DistanceMatrix[i, j] = DistanceMatrix[i - <span style="color:#a52a2a">1</span>, j - <span style="color:#a52a2a">1</span>];<br>                 <span style="color:#0000ff">else</span><br>                 {<br>                     <span style="color:#0000ff">int</span> a = DistanceMatrix[i - <span style="color:#a52a2a">1</span>, j] + <span style="color:#a52a2a">1</span>;<br>                     <span style="color:#0000ff">int</span> b = DistanceMatrix[i, j - <span style="color:#a52a2a">1</span>] + <span style="color:#a52a2a">1</span>;<br>                     <span style="color:#0000ff">int</span> c = DistanceMatrix[i - <span style="color:#a52a2a">1</span>, j - <span style="color:#a52a2a">1</span>] + <span style="color:#a52a2a">1</span>;<br>                     DistanceMatrix[i, j] = <span style="color:#2b91af">Math</span>. Min(<span style="color:#2b91af">Math</span>.Min(a,b),c);<br>                 }<br>             }<br> <br>         <span style="color:#0000ff">return</span> DistanceMatrix[valueOneLength-<span style="color:#a52a2a">1</span>, valueTwoLength-<span style="color:#a52a2a">1</span>];<br>     }   <br> <br> }</div> </div> </div> <p>and the test</p> <p></p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:9ce6104f-a9aa-4a17-a79f-3a39532ebf7c:1fa61020-f050-4976-8629-b3bcdc7f31e2" class="wlWriterEditableSmartContent"> <div style="border: #000080 1px solid; color: #000; font-family: 'Courier New', Courier, Monospace; font-size: 10pt"> <div style="background-color: #ffffff; max-height: 300px; overflow: auto; padding: 2px 5px; white-space: nowrap">[<span style="color:#2b91af">Fact</span>]<br> <span style="color:#0000ff">public</span> <span style="color:#0000ff">void</span> TestDistanceAlgorithm()<br> {<br>     <span style="color:#2b91af">LevenshteinDistance</span> x = <span style="color:#0000ff">new</span> <span style="color:#2b91af">LevenshteinDistance</span>();<br> <br>     Xunit.<span style="color:#2b91af">Assert</span>.Equal(<span style="color:#a52a2a">3</span>, x.Execute(<span style="color:#a31515">"Sunday"</span>, <span style="color:#a31515">"Saturday"</span>));<br>   <br> }</div> </div> </div> <p></p> <p>Tomorrow, I will post an blog entry explaining how you can use this algorithm to match exact or near exact data.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-5484348787313535682009-12-03T21:31:00.001-08:002009-12-08T09:30:40.160-08:00TDD for the DSL Parser…<p>Here I will document the TDD aspects of the parser diagramed in a previous post.</p> <p>I will also use this as a brief guide on how to do TDD in a real application environment.</p> <p>I will be using xUnit as the testing framework, and Gallio as the test runner. (also for sake of simplicity I am using C# since for some reason F# xUnit tests are not showing up properly in Gallio at the moment.)</p> <p>The empty test C# file is as below</p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:9ce6104f-a9aa-4a17-a79f-3a39532ebf7c:6352919d-5daa-40b6-9d77-0f061290813b" class="wlWriterEditableSmartContent"> <div style="border: #000080 1px solid; color: #000; font-family: 'Courier New', Courier, Monospace; font-size: 10pt"> <div style="background-color: #ffffff; max-height: 300px; overflow: auto; padding: 2px 5px; white-space: nowrap"><span style="color:#0000ff">using</span> System;<br> <span style="color:#0000ff">using</span> System.Collections.Generic;<br> <span style="color:#0000ff">using</span> System.Linq;<br> <span style="color:#0000ff">using</span> System.Text;<br> <span style="color:#0000ff">using</span> Xunit;<br> <br> <span style="color:#808080">///</span><span style="color:#008000"> Blog DSR Parser unit tests</span><br> <span style="color:#0000ff">namespace</span> blogParserTests<br> {<br>     <span style="color:#0000ff">public</span> <span style="color:#0000ff">class</span> <span style="color:#2b91af">ParserTests</span><br>     {<br> <br>         <span style="color:#008000">// Setup or Teardown tests in constructor or destructor</span><br> <br>         <span style="color:#0000ff">public</span> ParserTests()<br>         { }<br> <br>         ~ParserTests()<br>         { }<br> <br> <br>         <span style="color:#008000">// Tests Start here</span><br> <br>         <span style="color:#008000">// Test batch for Token Class</span><br> <br>         <span style="color:#008000">// Test batch for ScanBuffer Class</span><br> <br>         <span style="color:#008000">// Test batch for Scanner Class</span><br> <br>         <span style="color:#008000">// Test batch for Parser Class</span><br> <br> <br>     }<br>     <br> }</div> </div> </div> <p>I am ordering the tests in bottom call stack order, so that all low level classes are tested prior to higher level ones. This is merely my preference for when I look at test results in Gallio or the GUI of xUnit.</p> <p>Next up its time to put in a few tests, I normally only put in essential tests for the results of method calls and events when writing unit tests. The only time I test other things is if I need to test performance of some code or if I have some business login/formulas that need to be confirmed as working as defined by a client. </p> <p>Client driven unit tests is normally done when I have a story to implement that does not have a UI but does have a function in a UI or some important back-end system (hidden from the user) processing. This is one of the cleanest ways to get the processing accepted for a story in an Scrum sprint.</p> <p>OK back to the tests.</p> <p>When I write tests I always do it in a simple format. An example test is shown below.</p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:9ce6104f-a9aa-4a17-a79f-3a39532ebf7c:ea671067-743c-4e83-9b1b-f2b4d6d9b504" class="wlWriterEditableSmartContent"> <div style="border: #000080 1px solid; color: #000; font-family: 'Courier New', Courier, Monospace; font-size: 10pt"> <div style="background-color: #ffffff; max-height: 300px; overflow: auto; padding: 2px 5px; white-space: nowrap"><span style="color:#808080">///</span><span style="color:#008000"> Create a simple token from a sample ScanBuffer</span><br> [<span style="color:#2b91af">Fact</span>]<br> <span style="color:#0000ff">public</span> <span style="color:#0000ff">void</span> CreateSimpleToken()<br> {<br>     <span style="color:#008000">// Orchestrate Block</span><br> <br>     <span style="color:#008000">// Activity Block</span><br> <br>     <span style="color:#008000">// Assertion Block</span><br> <br>     <span style="color:#808080">///</span><span style="color:#008000"> Make test fail to start with</span><br>     <span style="color:#2b91af">Assert</span>.True(<span style="color:#0000ff">false</span>); <br> }</div> </div> </div> <p>When I code a single unit test I split it into 3 blocks of code or steps.</p> <ul> <li>Orchestrate – This is where is setup any variables/objects/data that is needed to perform the specific testing I am doing. </li> <li>Activity – This is where I do something that returns data or modifies Orchestrated data as part of the test </li> <li>Assertion – This is where I test assertions against data returned or modified in the activity block of the test. </li> </ul> <p>Also when I first create a test, I set it to always fail by default as you can see in the example.</p> <p>In xUnit you define a test by using the [Fact] attribute.</p> <p>In Gallio the test looks and fails as shown in the screen shot below.</p> <p><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="Gallio - DSRParser Test" border="0" alt="Gallio - DSRParser Test" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3ACFwf39LHQge2B9ki_FaSTJR-ERsxSqcXBDHau4kgGJM_WPUBdRXJIjSuD7jROkSEr5DCnTIfXesmSRonGyWt9_9FxhDzccXlYvAQu4JLpQtZr81YoL63Pl6cRoNs80ChuZ2Vw/?imgmax=800" width="457" height="418" /></p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:8eb9d37f-1541-4f29-b6f4-1eea890d4876:5208f90a-3815-4ee5-85f5-6c7daec1e5fb" class="wlWriterEditableSmartContent"><div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:8eb9d37f-1541-4f29-b6f4-1eea890d4876:5208f90a-3815-4ee5-85f5-6c7daec1e5fb" class="wlWriterSmartContent"></div> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-13395291152639703532009-12-03T21:11:00.001-08:002009-12-08T09:45:37.166-08:00DSL – Simple Parsing and Scanning Architecture…<p>In this blog entry I will revisit the DSL that I mentioned in earlier blogs. Except this time I will start to document parts of a simplistic interpreter that could be used as the basis of a DSL.</p> <p>The DSL syntax will be pretty simplistic to start with. but before I get a head of myself. I should show what the basic high level architecture will look like. <img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="ParsingPackage" border="0" alt="ParsingPackage" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh06Wnv8yvQfvICUGUUBITY-lTvsN9cpwphrathALdOaprb8EqjSJb6RdThalP2rxQF-p5Mw-LMzWoS8qb2zNaFQfNkVxq88_G6PPHyRQiAT5bUZTEbnZpEdoXbD1FETD0l88c0vg/?imgmax=800" width="466" height="322" /></p> <p></p> <p> <div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:8eb9d37f-1541-4f29-b6f4-1eea890d4876:2987033a-7ab2-4cd7-8ca8-84620f220ce3" class="wlWriterEditableSmartContent"><div style="padding-bottom: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; float: none; padding-top: 0px" id="scid:8eb9d37f-1541-4f29-b6f4-1eea890d4876:2987033a-7ab2-4cd7-8ca8-84620f220ce3" class="wlWriterSmartContent"> <lost end of this post will redo it in the future!></p></div> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-77749635140603753482009-12-03T15:11:00.001-08:002009-12-03T15:11:09.961-08:00Adaptive Learning Software Systems…<p>AS mentioned in the preceding post, I have worked on various learning systems in the past. By learning system I mean a technology that learns and adapts to its environment.</p> <p>For example if a system can detect a re-occurring pattern or conditions, it can learn how to deal with that situation based on previous actions in a probabilistic or deterministic manner.</p> <p>Most learning systems are able to make for want of a better word, educated guesses at processing tasks, data generation/selection based on historical actions and normally the following three core processing models:</p> <li>Linguistic models </li> <li>Semantic models </li> <li>Statistical models</li> <p>Each of these models stores, tracks and is evaluated based upon how the system or potential logic is used and applied.</p> <p>The models get quite complex, but are used extensively in many different applications, for example all three models are used extensively in speech recognition applications and VoIP IVR applications.</p> <p>These techniques can also be applied to almost all application types and are a natural progression beyond expert systems in many cases.</p> <p>I will do some blog entries later with examples of various models I have used in the past, plus some common public domain models that are available in the Open source community. </p> <p>It is a very interesting area, although gets very scientific very fast at times.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-54927101742366690272009-12-03T11:47:00.001-08:002009-12-03T11:47:49.765-08:00Adaptive Learning Patterns in MDM.<p>The Next post I am working on, is in this area. Not sure how much detail to go into just yet!</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-72087969647644439092009-12-02T16:51:00.001-08:002009-12-03T15:13:12.787-08:00Agile Product Management 101.<p>Alas, it is always painful dealing with product management that always changes requirements in near real-time.</p> <p>It is worse when product managers only give very vague requirements and either don't elaborate on the requirements in detail or worse create requirements on the fly as a system is being built and expect these requirements to be implemented as they emerge in a strange real-time like fashion.</p> <p>The above is quite normal for teams delivering ‘Green Field’ products/projects that start of as prototype technologies and are initially vague at best until certain technology concepts have been proven.</p> <p>Now the best way to deal with the above scenario in an Agile manner. At a high level is to take the following very basic steps.</p> <ul> <li>Create a high level feature list <ul> <li>Normally from any source that the Product Owner/Manager has access to. </li> </ul> </li> <li>Decompose high business value items into a little more detail. Good to involve an implementation team to do this. </li> <li>Get the Implementation team to say what it feels it can implement and demonstrate effectively as implemented within a time box. I prefer every 2 or 3 calendar weeks. </li> <li>Do not interrupt the implementation team as they deliver what they committed to. <ul> <li>If a new requirement is discovered save it til after the time box is complete and the team has delivered its [promise or not. </li> </ul> </li> <li>Evaluate implementation teams deliverable via a retrospective meeting <ul> <li>Highlight anything that is wrong </li> <li>Highlight the good </li> <li>Be constructive </li> </ul> </li> <li>Plan what will be delivered in the next time box </li> </ul> <p>Rince and Repeat.</p> <p>Now the key here is how you manage change and deadlines. </p> <p>A few key things that affect deadlines follow:</p> <ul> <li>HR related <ul> <li>Illness </li> <li>Personal Issues </li> <li>New Staff Hires </li> <li>Politics </li> <li>Staff Skill limitations </li> </ul> </li> <li>Technology related – issues in working with technology to implement a feature in a system for example. <ul> <li>Bugs </li> <li>Technology limitations </li> </ul> </li> <li>Business Related <ul> <li>Budget changes </li> <li>Custom Demo’s for sales/pre-sales/etc. </li> </ul> </li> <li>Added requirements, Missed requirements <ul> <li>Never add this to a running time-box </li> </ul> </li> <li>The list goes on </li> </ul> <p> </p> <p>There will always be commercial deadlines but this is where compromise comes in. </p> <p>Compromises can be any of the following:</p> <ul> <li>Add a requirement by dropping another </li> <li>Stop current time-box and re-plan a new one </li> </ul> <p>Never add features to an existing implementation teams committed work for a time-box.</p> <p>It is always better to stop a implementation teams work, and re-plan a new time-box of work team can commit to, Rather than trying to force a team to deliver.</p> <p>Always, demand to see a demo of what was promised by an implementation team at the end of a time-boxed implementation</p> <p>Never act as a project manager when you are a product manager.</p> <p>Never act as a Scrum Master when you are a product manager.</p> <p>Always be available  to the implementation team if they have questions on the product requirements they are working on.</p> <p>always have a retrospective meeting reference the time-boxed implementation</p> <p>Always look for good and bad in a time boxed implementation.</p> <p>Always respect the implementation team, never demand, ask and discuss/negotiate.</p> <p>You know the product and the business, the implementation team knows the technology and what can be done to implement the product. Both sides have to listen to each other as an equal.</p> <p>That's it for now.</p> <p><strong>There are plenty of examples of the above on google if you search for “Scrum Product Management” and “Scrum Product Owner”.</strong></p> <p>I will do a more defined and clear guide to agile product management later.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-39344357798123946782009-12-01T11:13:00.001-08:002009-12-01T11:13:07.969-08:00Very Large Transaction systems…<p>Well for the past year I have been working on a data management product that can deal with a ridiculous amount of data and data transactions.</p> <p>The product is a finance industry related product and is designed to work with Financial Market Data and also financial reference data feeds. Some of which can potentially have multi-million entries per file in some extreme cases.</p> <p>And of course the more feeds you process and match-merge the amount of data transactions (insert, update, removal, indexing, relational mapping and querying) can mean you process exponentially more data.</p> <p>Now the issue is simply there is only so much processing that can be done at certain parts of a transactional process before you hit some pretty major latency issues and processor performance issues, shortly after these are resolved you tend to hit physical infrastructure issues.</p> <p>A casing point is using SQL server bulkcopy on an optimized table you can import data at an incredibly speed, however if you need to process that data, mark it up in some way and merge it with other data, performance will nose dive. Add in basic auditing/logging support it will drop again.</p> <p>Most people resort to two main architectures in this light</p> <ul> <li><strong>BRBR – Bloody Row by Row Processing</strong></li> <ul> <li>This is the most flexible way of processing large data and provides many ways to improve speed, but is still along way behind batch processing in all but a couple of scenarios.</li> <li>BRBR can be optimized the following ways</li> <ul> <li>Multi-threading</li> <li>Parallel Processing</li> <li>Multi-database server</li> <li>Cloud and Grid computing and processing</li> <ul> <li>The above is pretty new, and not that mature yet</li> </ul> <li>Custom Data storage and retrieval based up data and how it is obtained</li> <ul> <li>There are a few of these out there including a few very powerful and fast data management and ETL like tools</li> </ul> </ul> <li>BRBR can suffer greatly based upon</li> <ul> <li>Skill of engineers</li> <li>Database technology</li> <li>Database latency</li> <li>Network latency</li> <li>Development tool used</li> </ul> </ul> <li><strong>Batch Processing</strong></li> <ul> <li>If you are able to just use a single database server you can look at the various ways of doing batch processing</li> <ul> <li>Batch processing can be Optimized by</li> <ul> <li>Having a great god like DBA</li> <li>Database tool selection</li> <li>Good Cloud/grid computing support</li> </ul> <li>batch Processing can suffer greatly </li> <ul> <li>Anytime you need to audit data processing that is done in batch data. most of the time you may have to run another batch process or a BRBR process to create audit data that can be generated easier at point of change in a BRBR based system</li> <li>Database dependency</li> <li>System upgrades</li> <li>Visibility into data processing/compliance</li> </ul> </ul> </ul> </ul> <p>That's it for now kind of a messy post. </p> <p>I will clean this up later, and add a post or two on advances in cloud computing in relation to Very large database processing issues.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-49023392561075331932009-11-30T11:09:00.001-08:002009-11-30T11:57:24.747-08:00A simple tool for Data Manipulation Projects…<p>When I am working on any kind of project that requires data manipulation at a database level. I normally have my teams create a simple data tracking UI in windows/of Linux that polls a table for changes and refreshes if any are detected. </p> <p>This tool is sometimes extended to prepare and view data used in demo systems.</p> <p>Since this is a commonly created tool, I am considering creating a more complex version of this utility for general consumption. I am also toying with the idea of attempting to use F# with either silver light or WPF to create the UI and UI logic.</p> <p>Idea’s on features for this tool would be useful.</p> <p>I will document initial very high level UI/UX design of the tool in later blog entries. As well as some key User stories.</p> <p>Right now the high level bullet list features are:</p> <ol> <li>List of tables being tracked </li> <li>View of data in table </li> <li>Simple way to save query against table if looking for certain data – simple SQL where </li> <li>Simple way to show related data – 1 level deep only </li> <li>Simple mechanism to refresh data based on a duration say every 5 seconds </li> <li>Simple way to show if a table has received an update -  for now just total row count(*) changes are enough. </li> <li>Ability to point to any database server (initially just SQL server 2005+) </li> <li>Windows based UI (WPF first, then maybe a silver light version) </li> </ol> <p>That’s all for now.</p> <p>The use case for the tool is simple, when I apply logic to update a table that has no UI, I can use this tool to show/demonstrate changes to data.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-70221205082625350922009-11-30T02:03:00.001-08:002009-12-08T09:44:45.361-08:00DSL 101 – Domain Specific Languages<p>Today, I spent quite a while helping a friend to architect at high-level a DSL (Domain Specific Language). </p> <p>Even though at the time he did not realize what he wanted to do was create a DSL. TO him all he needed was a way to do excel like formula expressions inside his code, but in a more English like syntax well as English as a chemical scientist can be I guess.</p> <p>To this end, I thought it might be worthwhile explaining what a DSR is, since I will build a few of them for the project’s I mentioned that I am playing with for fun on this blog as I learn F#.</p> <p><strong>So just what is a DSL:</strong></p> <p>A domain-specific language is a programming/specification language customized to a particular problem domain. </p> <p>This is quite an old concept as special purpose languages for modeling have always existed but recently have had a surge of emergence once again due to most newly created systems needing a more flexible way to extend logic used without the need of a programmer.</p> <p>Creating a DSL and the tools required to support it can be worthwhile if the language allows a particular type of problem to be expressed more clearly than existing languages would allow. An example of this is the issue of applying simple logic used to route a document for approval in a document management system.</p> <p>Another example of the new generation of DSL’s are products/tools like cucumber (from the Ruby world to do BDD), and in the F#/C# world we have tools that are DSL' based such as NaturalSpec (also a BDD tool this time for F#).</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-5116289516467697642009-11-30T01:35:00.001-08:002009-11-30T11:56:06.282-08:00EDM Councils Semantics Repository – Draft is looking good!<p>Well as part of my learning in the domain of Finance most specifically data repository and schema standards.</p> <p>As some people know I was and still am quit involved in CMM/CMMI and it was a pleasant surprise to find out that EDM council and Carnegie Mellon University have come up with a proposed data management maturity model focused on the finance industry as a whole. </p> <p>This standard in the making is based on the maturity models used in CMMI and PMM, and is split into 4 Distinct areas as follows:</p> <p><strong>Core Areas of DATA Management Maturity</strong></p> <p><strong>Operations</strong> <br />        Data Policies and Procedures <br />        Requirements Capture <br />        Vendor Contract Management <br />        Business precedence and data validation rules <br />        Data distribution <br />        Entitlement and authorization <br />        Archive, retention and privacy <br />        Business process and workflow <br />        links and associations of data and process <br />        SLA and resolution tracking <br />        Regulatory and client compliance <br />        Mapping and X-Referencing <br />        Extensibility and RE-use</p> <p><strong>Quality Management <br /></strong>        Quality Measurement and benchmarks (metrics) <br />        Classification (grouping/identity of metrics) <br />        Change and Exception Management <br />        Error Logging and Root Cause analysis <br />        Inventory (data), traceability and surveillance <br />        Cleansing, enrichment and validation <br />        Quality assurance and auditing <br />        Data quality strategy and objectives</p> <p><strong>Governance & Strategy</strong> <br />    Strategy, goals and scope <br />    Data content and coverage <br />    Funding <br />    Executive support/sponsorship <br />    Governance operating/org model/structure <br />    Communications/Internal marketing/PR</p> <p><strong>Platform</strong> <br />    Metadata repo <br />    Legacy reconciliation <br />    Architectural framework    <br />    Data repo standardization <br />    Transformation logic <br />    Format standards and messaging model <br />    semantics and definitions <br />    Loading and application integration <br />    common data model</p> <p>As you can see the Maturity model is focused on 4 key area’s that are key to data management in the finance industry.</p> <p>Also of special note is the Semantics Repository that has been a work in progress for quite a while now, but seems to be nearing completion.</p> <p>The draft specification, excel sheets and diagrams are at the flowing link: <a title="http://www.hypercube.co.uk/edmcouncil/" href="http://www.hypercube.co.uk/edmcouncil/">http://www.hypercube.co.uk/edmcouncil/</a></p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-86171970776023596862009-11-27T13:59:00.001-08:002009-11-30T11:55:03.025-08:00What is the definition of Done.<p>Seems I am being asked this question a lot in agile groups all of sudden. So lets see if I can put my definition on paper as it were.</p> <p>Ok first and foremost I hate percentages, the reason for this is that they really mean nothing when it come to determining when a project can be delivered or not. </p> <p>A good example of this is the team member that always says that his work is almost done, and when you push its 80% or 90% done. Yet a few days later it is still 80-90% done. To me work is either 100% done or 0% not done, there is no in between.</p> <p>Now my view percentages is out of the way, what do I term as done in Scrum or indeed any development methodology.</p> <p>That is quite simple a piece of work is done if it can be demonstrated as working according to its requirement/feature definition or User story. It is confirmed as being done if the product manager signs it off as being done. Most requirements should have a description of the feature, a description of how to test it, and ideally if its a code based deliverable – unit tests proving it works at system level for any output of the code/api based on known inputs and expected good/bad results. It can also include documentation artifacts as well.</p> <p>Now next comes the catch, the definition of done should be defined by the team doing the work and the Product Manager if one is available. By no means is the list above exhaustive nor is it the minimum of what should be done. However it is the minimum I  would like done when running an agile team that understands TDD.</p> <p>I love unit testing, it saves just so much time… ok I digress…</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-31536582805260176262009-11-27T13:30:00.001-08:002009-11-30T11:54:41.126-08:00The Enabling Requirements Document…<p>We have all worked in environments where either Product managers or delivery heads in a company give high-level vague requirements. But for some unknown reason expect engineering / Development / Implementation teams to deliver tangible results.</p> <p>In-fact this is a subject I am talking to a fair few people about in the Agile communities I frequent, most notably the Scrum based ones. </p> <p>During discussion someone asked what is a good definition or measure that can be used to day if a requirement was suitably defined. After some thought I come up with the following definition, but as usual I have some pre-conditions that help out, so here it goes: </p> <blockquote> <p><b>"A Requirement specification is good if it allows a person of average skills to implement the requirement without experimentation."</b></p> </blockquote> <p>This means that if the requirement is not clear and needs to be explained more, or if the skills of the team implementing the requirement are not sufficient for the task. There is a great likely hood that any understanding of the requirement may well be incorrect.</p> <p>The best requirement specification should enable a team with the right skills to implement it. However, getting to this level of maturity in product requirements/feature lists/backlogs can be hard. Also it needs teams to be realistic about what skills they do and do not have, and be able to articulate that will need time to research/be trained/mentored in skills required. Or even articulate that the requirement is just not good enough to work on due to its vagueness.</p> <p>User stories help a little in this, but for a feature from a product/business focus a definition/specification needs to be created that is not vague and can be used as a basis for story creation.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-77792189340398963472009-11-27T02:06:00.001-08:002009-11-30T11:53:54.960-08:00System Based Workflow Vs. Process Based Workflow…<p>There are two types of workflow that come to mind when people talk about workflow systems in software. Most of the time it seems that they get confused with each other since most people just use the term workflow to mean either of these types of workflow.</p> <p>To clarify this, I decided to put down my views on what they are to me:</p> <ul> <li><strong>System Based Workflow</strong> – A application level workflow system that executes processing logic in a sequence or based on a state in the system that has changed. A common Microsoft based workflow system of this type is Windows Workflow Foundation. (WFF) </li> <li><strong>Process Based Workflow</strong> – A software level workflow or to be more accurate approval system. This type of workflow is used to approve changes in document management by routing update/changes/requests to other users in a system for approval or rejection or have the system automate the approval/rejection based on user defined logic. Document management and Content Management systems use this kind of workflow all of the time when it comes to publishing content/data. It is also the type of workflow used to implement 4-Eyes like maker/checker logic in software application. </li> </ul> <p>Not sure if that helped, but there you have it.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-47389416258895886632009-11-27T01:30:00.001-08:002009-11-30T11:53:24.683-08:00About the Workflow system I am creating in this Blog…<p>Ok now it’s time to explain what I mean when I talk about creating a simple workflow system for this blog.</p> <p>Without going into too much detail there are two main types of workflow system:</p> <ul> <li><strong>Sequential Based Workflow</strong> – This is where Activities are executed one after another based on the results of a previous activities end state. </li> <li><strong>State Base Workflow</strong> – This is where activities are executed based up a system state or event, they do not run in a logical sequence but more react to what happens inside a system. </li> </ul> <p>The system I am creating for this blog will initially be a sequential based one, before i get to specifics it is better to detail the parts of a workflow system so here goes…</p> <p>At a high level there are quite a few parts to a workflow these are listed below:</p> <ul> <li><strong>Workflow system – </strong>The glue and process that enables all processing done in a workflow. </li> <li><strong>Workflow Activity Map</strong> -  A collection of activities that has a natural start and end point with activities chained one after another in sequence. Workflow activity maps sometimes have embedded logic that helps the system select the next activity to process in a sequential workflow model. or just a collection of activities that have Zero or more dependencies in a state based workflow system). </li> <li><strong>Workflow Activities</strong> – The actually activities that perform some process and have a return value or result. Activities also can point to the next logical activity. </li> <li><strong>Workflow Workers</strong> – The heart of a workflow system is the worker. the worker is a token/object/item that contains context data related to what is being processed by the workflow activities in a workflow. </li> <li><strong>Workflow States – </strong>only really useful in a state based workflow. This is a list of states that are available to be monitored by a state based workflow’s activities. (I will go into this more, when I create a simple state based workflow solution for this blog.) </li> </ul> <p><strong>A little more on Workflow Maps.</strong></p> <p>As stated above the workflow system I will create on this blog is a sequential based one that is a series of <strong>ACTIVITIES</strong> than are processed sequentially in a <strong>Workflow Map</strong> as shown below:</p> <p><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsOdgd8cW24Hj3dWacIbyvrJBL-y8Y7CAjEhvEVeBLAEo937jywDECEGJN7s6hhFFZcX4m8Bff9KIFGI2pdUyH6mbrFKHedYWh5lkc5x5MmAab3SPINpzKIUT-2oioz4cgRrbtlw/?imgmax=800" width="436" height="38" /></p> <p>All sequential workflow systems have a start point where the workflow starts which basically just points to the first activity to process.</p> <p>Once that activity has completed processing it will then move onto the next activity, Until it reaches the end of the workflow map.</p> <p>Well I think that is enough for now, In a later post I wil explain how and where <strong>workflow workers</strong> come into play and then detail the types of activity that can be in a workflow system, and then detail which ones I plan to support in the system I am creating for this blog. </p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0tag:blogger.com,1999:blog-3594539.post-1825253610136628802009-11-27T01:05:00.001-08:002009-11-30T11:52:58.995-08:00An issue of Context…<p>When creating a rule/logic/workflow engine there are various things that need to be considered in it’s architecture.</p> <p>One of the most overlooked aspects of such systems is that of the <strong>CONTEXT</strong> in which a rule is run, and how it affects what a rule can do.</p> <p>To elaborate a little more, rules normally know nothing of the environment that they run in, rules just execute logic based upon data that should be available when the rule is evaluated for its result. (a result is normally a logical true/false, but it can be other values and data types depending on its usage and context).</p> <p>So what do I mean by Context, quite simple the context for an executing rule is the environment that it runs in. </p> <p>A context normally defines what data is available to a rule for example:</p> <ul> <ul> <li>Global Values </li> <li>Results of nested logic that is used in the rule </li> <li>System data that is applicable to the rule being evaluated </li> <li>Functions or other rules that can be called by the rule </li> </ul> </ul> <p>A rules engine when it is instantiated/run should go through the following processing steps:</p> <ul> <li>Setup context for the rule (normally defined by a flag or driven by a workflow system) </li> <li>Execute a rule or batch of rules in context </li> <li>Tear down context (sometimes a context is persisted depending on the type of rule logic and system it is in) </li> </ul> <p>Later on, I will be creating a simple logic/rule engine that conforms to the above and deals with the issues in creating and synchronizing a context and its data.</p> <p>I hope this has been at least a little helpful, soon the real fun and more technical stuff will be started.</p> Scott Worleyhttp://www.blogger.com/profile/14856617555077085190noreply@blogger.com0