Sunday, May 24, 2015

jcLSL - A very lightweight parser (soon to be a scripting language)

Intro

Yesterday I posted about my GitHub repo work over the last couple of weeks. One project I did not include intentionally was jcLSL. Followers of my blog know I had been toying with writing my own scripting language for quite some time, but kept hitting road blocks in parsing and handling anything some what complex. This has led me to think about what I would really want in a scripting language and what value it would bring to the already feature rich C# eco-system that I live and work in every day.

The Problem

At the end of the day I found that I really would only want a scripting language for doing mail merges on strings. Expanding on that, how many times have you needed to have a base template and then populate with a class object or even simply some business logic applied? I'm sure we've all done something like:

public string ParseString(string sourceString, string stringToReplace, string stringValue) {
	return sourceString.Replace(stringToReplace, stringValue);
}
While at a simplistic level this is acceptable and a better approach than not wrapping the Replace call, it isn't ideal especially if you are working with POCO (Plain Old CLR Objects), then it becomes a lot more dirty in your code, assuming you wrapped the calls like in the above function, let's say you have a basic User class definition like so:

public partial class User {
   public int ID { get; set; }
 
   public string Name { get; set; }
}
And then in your parsing code assuming it also exists in a User class:

 
public string ToParsedString(string sourceString) {
   sourceString = sourceString.Parse("ID", this.ID);
   sourceString = sourceString.Parse("Name", this.Name);

   return sourceString;
}
As you could guess not only is that code an eye sore, it doesn't scale when more properties are added to your class and you'd end up adding something similar to each POCO in your code base - not ideal. This brought me to my first objective, solving this problem in an extremely clean fashion.

Earlier this week I got it to where any POCO with jcLSLMemberAttribute decorated above a property will be automatically parsed handling the situation of code changing in time (some properties could go away, be renamed or added). With my current implementation all you need is to define a jcLSLGenericParser and then call the Run method passing in the string to mail merge and the class object you wish to merge from like so:

var TEST_STRING = "Hello {Name} This is a test of awesomness with User ID #{ID}";
var user = new User { ID = 1, Name = "Testing" };

var gParser = new jcLSLGenericParser();

var parsedString = gParser.Run(TEST_STRING, user);
After running that block of code the parsedString will contain: Hello Testing This is a test of awesomeness with User ID #1. There is an optional event for the jcLSLGenericParser class as well if more custom parsing needs to be achieved.

Going forward

One of the first things to do is to add some more options for different scenarios. Maybe you've got a large code base and going through and adding the jcLSLMemberAttribute decoration would be a task in itself. One approach to solve this scenario is to add an optional parameter to the jcLSLGenericParser constructor to simply iterate through all the properties. This takes a performance hit as one would be expect, but it would leave the level of ties to this library to a minimum. Thoughts on this, please post a comment.

On the larger scale my next major goal is to add support for Streams and other output options. Let's say you had a static HTML template that need to be populated with a News Post for instance. The HTML template could be read in, mail merged against the database and then outputed to a string, stream output to a file or return binary. Trying to handle ever scenario I don't think is realistic or feasible, but handling the 90% scenario or better is my goal.

Another item that I hate implementing with mail merge fields is error handling. One approach is to simply return the exception or error in the mail merge itself. This isn't good when you have external customers and they see something like Object Reference Exception or worse yet a full stack trace. The vailidity of your product will go down quickly. However, I think a standardized {JCLSL_ERROR} merge field to store the error to display on an exception page or email would make sense. Handling both objectives in error handling: being aware of the error, but also handling it gracefully.

Further down the road I hope to start adding in support for "true" scripting so you could have conditionals within the merges or other logic you would want to be able to change on the fly without having to deploy an updated Mobile App, ASP.NET MVC Web App, WebAPI Service or whatever platform you're using with this library.

Where to get it

As mentioned earlier you can grab the code and samples on GitHub or grab the binary from either NuGet or from the NuGet Console with: PM> Install-Package jcLSL. As I check code into GitHub and get to stable releases, I'll update the NuGet package.


Saturday, May 23, 2015

May 2015 Open Source GitHub Project Updates

While largely quiet on here since my last post two weeks ago I have been hard at work on several smaller projects all of which are on GitHub. As mentioned previously, everything I work on in my freetime will be open sourced under the MIT License.

jcAnalytics

The first item I should mention is some new functionality in my jcAnalytics library. Earlier this week I had some ideas for reducing collections of arbitrary data down to distinct elements. For example if you had 3 Objects of data, with 2 of them being identical, my reduction extension methods would return 2 instead of 3. This is one the biggest problems I find when analyzing data for aggregation or simply reporting, especially when the original amount of data is several hundreds of thousands or more. I attempted the more straight forward single threaded model, as expected the performance as the number of elements increased was dramatically slower than a parallel approach. Wondering if there were any theories on taking a sampling of data quickly to scale as the number of items increased, I was surprised there was not more research on this subject. Doing a Log(n) sample size seemed to be the "goto" method, but I could not find any evidence to support the claim. This is where I think recording patterns of data and then persisting those patterns could actually achieve this goal. Since every problem is unique and every dataset over time the extension methods could in fact learn something along the lines of "I have a collection of 500,000 Addresses, last 10 times I ran I only found 25,000 unique addresses at an average rate of every 4 records." On subseqent runs, it could adapt per request. Maybe assign Guids or another unique identifier for each run with the result patterns on disk, in a SQL database or in Azure Cache. For those curious, I did update the NuGet package as well with these new extension methods. You can download the compiled NuGet Package here on NuGet or via NuGet Console with PM> Install-Package jcANALYTICS.Lib.

jcPIOL

A huge topic in my world at work has been offline/online hybrid mobile applications. The idea that one could "sync" and then pull down data for 100% offline use has been on my mind since it was requested several months ago by one of our clients. Knowing the first approach might not be the best and that I wanted to create a generic portable class library that could be plugged into any mobile application on any platform (iOS, Android, Windows), I figured I would begin my research fully exposed on GitHub and then as stable releases were built I would publish them on NuGet. This project is of a larger nature in that it could quickly blossum into a framework instead of simply a library. As of right now on GitHub I have a the GET, POST and DELETE HTTP verbs working to pull/push data, but not storing the data for offline purposes. I'm still working out the logistics of how I want to achieve everything, but the ultimate goal would be to have any request queued when offline and then when a network connection was made automatically sync data. Handling multiple versions of data is a big question. Hypothetical if you edited a piece of information and then edited it again, should it send the request twice or once? If you were online it would have sent it twice and in some cases you would want the full audit trail (as I do in the large enterprise platform at work). Another question that I have not come up with a great answer for is the source of truth question. If you make an edit, then come online I could see a potential race condition of the data syncing back and a request being made on the same data. Handling the push and pull properly will take some extensive logic and more than likely might be a global option or down to the request type level. I am hoping to have an early alpha of this working perfectly in the coming weeks.

jcTRENDNET

This project came at the request of my wife who wanted a way to view Trendnet cameras from her Nokia Lumia 1020 Windows Phone. Trendnet only offered apps for iOS and Android and there were no free apps available in the Windows Phone marketplace - so I spent an evening and wrote one last August (2014). Again going with the Windows 10 Universal approach, I began to re-write the app to take advantage of all the new XAML and addin features I had long since wanted to add in. Going with my open source initiative, all of the code is checked into GitHub. I am hoping to have everything ported from the old Windows Phone 8.1 app along with all of the new functionality this summer.

jcRSS

Another older project that I see a need to fufill going forward. Since Google Reader faded away, I switched over to feedly, but I really don't like their interface nor how slow it is. Originally this project was going to be an ASP.NET MVC/WebAPI project with a Windows Phone/Windows Store app. As with my other projects, I knew I wanted to simply port over the work I had done to a Windows 10 Universal App, but as I got into working on it, there was no reason to tie the apps back to a WebAPI Service if I did away with the MVC view. Knowing I was going to be freely giving away this application and didn't want to have ads I also didn't want to incur massive Azure fees if this were to take off. So for the time being this project will exist as a Windows 10 Univeral App with full support for multiple devices (i.e. if you read an article on one device, it will mark it as read on the others). You can check out the code on GitHub. I'm hoping for a release in the coming months.

Federation

This was a project I had been slowly designing in my head since summer of 2012 - a turn based Star Trek game without microtransactions and the ability for one to simply keep playing as long as they want. I started coding this in August 2014 and into September 2014, but put it on hold to work on Windows IoT among other topics of interest. Now with Windows 10's release on the immediate horizon I figured I should wrap up the game and in kind open source the project. As of now I'm in the process porting over the XAML to Windows 10 as it was originally targeting Windows Phone 8.1. Once that process is complete, I will return to working on the logic and with any luck release it sometime this summer, but in the meantime you can checkout the code on GitHub.

jcMATH

I originally wrote this "game" for my boss's child since there was not a dot math game in the Windows Phone marketplace. Seeing as how it got 0 downloads, I open sourced it. I did start porting it over to a Windows 10 Universal Application, but have not finished yet.

Closing

Now that Visual Studio 2015 RC is out, I will more than likely be returning to my open source bbXP project. The only reason I put it on hold was the issues I was running into with NuGet packages in CTP6 of Visual Studio 2015. Coming up in a few weeks is the 20th anniversary of when I wrote my first line of code, expect a retrospective post on that in a few weeks.


Sunday, May 10, 2015

Silicon Graphics Onyx 2 Restoration

Silicon Graphics Onyx2

Intro

Those following my blog for some time know my passion for Silicon Graphics machines. After having picked up a Silicon Graphics Onyx 2 last fall I finally had some time to get the "big iron" up and running. The Onyx 2 is interesting in that it is the one of the last "board" based deskside graphics machines. Following the Onyx2 the only comparable workstation would be the Tezro offering up to 4 R16000 1ghz CPUs and a 128mb V12 Odyssey graphics card. My specific Onyx2 CPU wise was nearly maxed out with 4 400mhz R12000 MIPS cpus and completely maxed out in ram at 8gb. Graphics systems wise it was pretty low end. It came with a DG5-2, GE14-2 and 1 RM8-16 Raster Manager, effectively making it a Reality graphics system. Fortunately eBay in time had a DG5-8, GE16-4 and 2 RM10-256 boards for extremely cheap so after swapping the boards I now have an InfiniteReality3 system. The InfiniteReality4 (the last generation) only differs by offering the RM11-1024 (1gb of texture memory) versus the 256mb per board I have in the RM10s in addition to Pixel Fill Rate differences of nearly double.

Silicon Graphics Onyx2 - GE14-2
Geometry Engine (GE) 14-2

Silicon Graphics Onyx2 - GE16-4
Geometry Engine (GE) 16-4

Silicon Graphics Onyx2 - RM8-16
Raster Manager (RM) 8-16

Silicon Graphics Onyx2 - RM10-256
Raster Manager (RM) 10-256

Like most of the machines I have gotten second hand they come with the original slower scsi drives. This Onyx2 came with the original 9gb IBM Ultra-Wide SCSI 2 hard drive with IRIX 6.5.8 on it. Knowing from the listing the cd-rom drive was faulty, I simply copied all of the IRIX 6.5.30 cds over NFS to upgrade it. After which like I had done with my Silicon Graphics Origin 300 back in 2012. For those inquiring I chose to my goto Ultra 320 SCSI Drive, the Maxtor Atlas 15K II. Silicon Graphics Onyx2 - Original Harddrive
Silicon Graphics Onyx2 - Replacement Harddrive

A look at my Onyx 2 all wired up, note this was before I swapped in the DG5-8:

Silicon Graphics Onyx2 - Back wired up

I should note anyone curious getting an Onyx2, you should keep it in a cool place or outside of a bedroom as the fans (which are temperature controlled) when in full speed are quite loud.

Benchmarks

Knowing one of the first things I do after getting a new system up and running is benchmarking it with my own cross-platform CPU benchmark, jcBENCH. Wanting to compare the R12000 and R14000 architectures, specifically with my goto 4xR14000 600mhz Silicon Graphics Origin 300 I ran jcBENCH. Surprisingly with the extra 200mhz (50% increase) and enhancements the R14000 MIPS cpu brought, my Origin 300 is over 3 times faster in both integer and floating point tests.

Since I had never had an InfiniteReality system before I wanted to test it with something semi-recent such as Quake 3. Knowing it was not optimized for gaming, let alone optimized for IRIX I was still intrigued.

For my Quake 3 benchmarks I used the neko_quake3-1.36.tardist release leaving everything on the highest settings except filtering which I left on bilinear. For each test in Quake 3 the only things I changed were the resolution and bit depth. No other processes were running.

Silicon Graphics Onyx2 - Quake 3 Benchmarks

The results were pretty interesting. Being on just a step down from the highest end cpu I figured the performance might actually be better with the InfiniteReality3 installed. If anyone reading this has an IR3 and 4xR14k Onyx2 please run the same tests and let me know your results. Overall the biggest jump was swapping the RM8-16 with an RM10-256, especially when using 32bit bit depth. What I found most interesting is the addition of a 2nd RM10-256 and swapping out the GE14-2 for a GE16-4 brought upon diminishing returns. This leads me to believe at that point Quake 3 became CPU limited with my 400mhz R12000s. Knowing that this particular build is single threaded I am curious how my 600mhz R14k Fuel with a V10 would perform in comparison (a test I will do in the coming weeks).

Closing Thoughts

For a machine that if bought new in May 2001 would cost $252536 (per this pricing sheet), I feel as though I have a piece of history that for a time blew away what was delivered by the PC and Mac worlds. Based on my own research comparing systems with PCs of the time (and other workstation manufacturers like DEC and Sun), the Onyx2 was one of the last extremely competitive offerings Silicon Graphics had. One could argue the Octane 2 was the last. Companies like 3dfx (interestingly enough had several Silicon Graphics employees) and AMD drove the PC industry forward with their Voodoo and Athlon products respectively; the "death of the workstation" so to speak.

Going forward with the Onyx 2 I hope to add some Onyx 2 specific optimizations to the ioquake3 project taking advantage of the Silicon Graphics OpenGL extensions that could speed up rendering. Along this path I would also focus on V10/V12 optimizations to bring Fuel and Tezro machines a more optimized experience.