Over the last nearly 8 years at Comcast as part of the ‘Online’ group and later as founding member of mighty Comcast Interactive Media I’ve made many business connections, met a number of excellent peers and forged a few relationships that will last a lifetime.

As a happily married 40+ dad with mortgage and college payments I often feel over the hill with regards to blogging and living out loud. As a Manager with a sometimes disproportional ego I also felt an obligation to ‘disappear’ for some time in order to give @tomjbarker room to make the team his own. I’ve no doubt that he will take what we started to the next level.

When I joined Comcast we had acquired a portal using technology from the last millennium (JSP, coded like it was 1999). It was hosted by a company we competed with and after 3 months I gave up trying to get a stable build from the source code. We merged with a development team who hired java test writers because ‘real’ developers don’t waste time writing unit tests. 2 key executives took this mess and in what I have to call a ‘visionary’ move decided to craft a ‘web development’ culture within Comcast. I’ll always be indebted to Sam Schwartz for doing whatever he did to convince Comcast that splitting off a wholly owned subsidiary as a web shop would be a good idea. That gave us the opportunity and Charlie Herrin was the one whose personality and direction shaped our day to day work.

The early days were exciting and we took an empty room and filled it with whirring servers powering a living architecture that grew up to power xfinity.com and xfinityTV.comcast.net (The architecture we used was this new thing called REST wrapping a key based XML document store). We took a team who didn’t understand testing in an untestable environment to a group passionate around TDD.  We introduced Java/Oracle biased executives to the wonders of S3, Node and Rails. Code bases with 0 tests now have thousands of meaningful ones. We went from developers oncall 24/7 and my phone ringing 3 nights a week to me sitting in the park with my kids as the software absorbed feed failures and gracefully degraded.

I was the architect (along with @kmartino) and a primary coder of the software stack and trained a team to support it. That team has surpassed me in every way as coders.  The level of individual talent is staggering.  A CIM jr engineer would be a Sr at any other company in Philly.

The business models of CIM were fairly mature and the products were very much in an incremental growth phase.

At this point I got a call from a headhunter looking for someone to lead a development team for an Energy company.  I was skeptical because I didn’t really get the new competitive energy market and didn’t really trust the companies selling it.  When I met some of the marketing folk they hooked with me with a simple pitch.  Nobody really knows how to market energy and Energy Plus is going to figure it out.  They have a business team that is incredibly agile and data driven with not an ego driven decision in sight.

As a new company they had much of their infrastructure built by waves of vendors who made a hash of their codebase.  They have made a real investment in IT to make us part of their competitive advantage and I’m looking forward to the challenges.  Right now companies build a plant and sell energy, in the future they will have to market it.  Much like Comcast formed CIM to figure out how to do web development, Energy Plus was acquired by NRG in order to figure out how to market energy.

They are doing business iterations and retrospectives in order to invest in what works and adjust what is not optimal.

Nothing hammered how good they are  into my head as much as my first meeting with our finance team.  They were asking for development help for some statistics they were compiling.  I’d heard they were trying to use python to download some data and analyze it in excel.  I walked in thinking I’m the big python expert and with a passing knowledge of ‘R‘ was going to be able to help them. They described the flow of downloading, parsing, processing, analyzing and explained it was a manual process done daily.  I asked how long it takes them thinking I’m about to save some poor human hours a day.  They said about 1omin since its all scripted.  They just have to run a batch file.  Wow, I know developers that don’t take time to automate like this.  The downloading is solid python analyzing is hardcore R.  They are professional enough to come to me to ‘operationalize’ this as the next step.  There really are no ‘B’ players here and when someone on the business side asks me for ‘help’ it is for real.

So in short the only thing keeping us from being fully agile is engineering, which means the only barrier to agile is ME getting it done.

And for this story I have no blockers

Usually I have to spend the first few minutes or an Art exhibition reading the background to understand the story that the gallery is telling through the art.  This one was different.

Rather then view the Mapplethorpe images through the lens of an expert Sean Kelly’s people found 50 Americans, one per state, to curate the selection.  They were nervous about what selections the wisdom of the crowd would choose to surface.  Some people involved with the Mapplethorpe foundation were initially skeptical of the project.

Proving once again trust and risk often yield success these 50 random Americans put together a selection of images that were stunning.  I was able to walk though the gallery, take in an image, and read a short selection about why that image was selected by an American.

The Americans ranged through every cross section I could think of, from successful businessmen (Owner of local favorite Dogfish) to a Rabbi to my lovely wife.

Mapplethorpe is often remembered for the controversy surrounding his images.  They have been called exploitive, indecent or simply obscene.  Surprisingly all I saw was crisp compelling photos with an extraordinary use of light and contrast.

“Experts” in academia has praised his artistic works.  ”Experts” in law and government have condemned his work.

Turns out average Americans just want the opportunity to enjoy his work.

I’m spending some time doing some technical spikes to see what is possible and I found a need to load a particular file from a git repo given the path and tree hash.

I grabbed my trusty python, dulwich (native python-git library) and gave it a shot. After a few minutes writing complicated looking recursive code I jumped over to irc where the friendly author pointed me to a convenience function that does what I needed.

Here is the short answer:

from dulwich.repo import Repo
from dulwich.object_store import tree_lookup_path

r = Repo('/Documents/projects/gitdep/rails')
def get_file(tree, path):
    (mode,sha) = tree_lookup_path(r.get_object,tree,path)
    return r[sha].data

tree = '7e7331fce169bbe1d6be71a30c1e1f7ab2e6ceba'
path = 'activemodel/examples/validations.rb'

print get_file(tree,path)

This gives me a rails validation file from last year. Nothing special about this file, I just find the rails git repo an interesting playground for git experimentation.

I’m definitely perceive an intangible benefit in culture and fun. With a full test suite and engaged developers working out loud coding is fun again.

I give a lot of thought to developer efficiency and generating metrics around our output is very important to me.  We are at a point in this iteration where I have too many stories in progress and it is taking a few extra days to get work completed and accepted.  This does not concern me greatly since this is a new team and it usually takes a few turns to get into a rhythm.  I was walking down the hall and one of developers said that things are going ‘Awesome’.  I said “Great, but awesome is not a metric“.

Since then he and the rest of the team have risen to the challenge and we have some demonstrable facts in low bug counts and high numbers of actual hours (hands on keyboard time) logged.

I’ll have a clue in a few weeks and ‘know’ in a few iterations that TDD and pair programming have raised productivity in this team.  For right now I do think things have improved and I feel that our velocity is going to increase.  IMHO my personal job satisfaction has increased.

So I may have been nieve last week in saying awesome is not a metric.  I’m now thinking that awesome must a metic with a correlation to both employee retention and code quality.  We simply have not developed the tools to measure and understand this thing we call ‘awesome’.  I can only observe the secondary effects that occur when there is “awesome” within a team.

This is kind of like gravity.  Science can only measure the effects of gravity but you can’t run without it.  And running beats floating back and forth aimlessly any day…..

I picked up a 4lb HP DM4 and dual booted to Ubuntu with not issues.  The trackpad didn’t work right, but a script off someones blog made it work reasonably ok.  A quick hop into debug mode showed that the drivers returned negative x-y coordinates when you use 2 fingers.  I downloaded the open source drivers to take a look at the code and saw that a patch was already in head.

Developing was a joy and eclipse opened nearly instantly.

Microsoft silverlight DRM is not (yet?) ported to linux so I dual boot to windows 7 to get the customer experience and use CIM products.  A funny thing happened to me while getting the ‘customer’ experience.   I liked it!   Windows 7 is much more usable then either osx or Linux and IE9 looks like it will be a really powerful platform for future development.

Around this time I picked up a python update on OSX and spent about 4 hours trying to get mysql and python to talk to each other.  I booted into windows and thought about how to use this platform as a development machine.

Enter BitNami. Rather then deal with version conflicts I grabbed a virtual machine that is close to my target server and installed VMware Player.  Mapping a windows directory to the virtual machine lets me edit files in native windows while running my build chain on Linux.  The browser I use for development is finally the same browser used by the majority of my customers and my dev environment is much closer to my server environment as well.

For not I’m working through the rough edges of this setup and trying to find a decent windows SSH client but this setup seems to have legs.

I’m used to reading about VIm plugins and tips on sites like http://www.vim.org/ but I never really thought about the power of collaborating on the ultimate VIm config via the forking, pushing and pulling that is github.  The idea of forking someones config and :%s/leader=’\'/leader=’`'/g is really compelling.

So I got to my next session, Philly Opendata, a bit early and powered up the University of the Arts guest wireless to see what this gitHub dotfile concept is really about and could not connect. After an hour of discussing how VIm keeps your hands on keyboard and head focused on the task I’m again reduced to a mouse clicking consumer wondering where the button is that will get my osx to work, or give me a clue as to why it won’t.  I would have searched for “iwspy on osx” but without wireless it would have been a short trip.

The point of this segue is that Maurice noticed my plight and leaned over with the security settings to his mifi app to get me out of the 80′s.   That has never happened at any formal show I’ve been to.  He had signal and was willing to share.

Turns out Philadelphia has data and is also willing to share.

Getting access to this data is important for so many reasons.  Before I talk about the social importance of opendata I have to step back and comment that crunching huge datasets and experimenting with bizarre visualizations is the type of fun that drew me to computers in the first place.  Whether plotting Fourier power transforms of breathing sheep or number of potholes per coffee shop there is something exciting about creating a unique perspective on otherwise boring or overwhelmingly complex data.

Having transparency into our government machine is really just a continuation of our need for independent newspapers.  Thomas Jefferson is often quoted as saying that Democracy depends upon an informed population.

The city of Philadelphia has committed to releasing 311, GIS and crime data. There was a very lively discussion at this session around the types of apps we could build, the benefits that the city will reap and most importantly WHEN it will launch.

This was all good and exciting conversation and one particular thread stuck with me long after the session ended.  We talked about how we can audit the data and ensure that it is used properly and not abused.

We live in an age when a comedian can save the elephants by editing a wiki page.  When I was a kid I was raised to believe that reporters were ‘investigative journalists’ and dug deep to get the facts.  I grew up just a few miles from where Geraldo Rivera got his start with an acclaimed expose of a mental institution that resulted in action being taken.  I was not until I got older and became exposed to Fox and the Daily show that I’d realized how seemingly sane people could look at the same data information and draw such dramatically different conclusions.  (Actually I learned that in grad school.  ”Number-smithing” and “creative graphing” are required classes for practicing engineers.  There was a similar class for the business students, but it skipped the numbers part altogether.)

It would be really easy for me to take all the pothole data and plot it on google maps.  A little massaging could show my street as having a few extra’s and maybe I’ll even ‘fix’ a few in the other neighborhoods.  People will see my app and maybe my street gets a little bump in priority.

Visualization is a powerful tool and data of this magnitude will always be analyzed with bias.  Even a simple flow chart describing the new health care policy can be warped towards an agenda.

Opendata puts us all on a level playing field.  I’m looking forward to the apps and visualizations coming out.  I will be disappointed if we wind up with a  liberal and a conservative app mirroring our polarized two party system.  I personally feel that there is a responsibility in the hands of the designers and developers to attack this data and ensure that the people of Philadelphia have multiple avenues of getting accurate facts.

The first tenant of the IEEE code of ethics hints at the damage we can do here.

1. to accept responsibility in making decisions consistent with the safety, health and welfare of the public, and to disclose promptly factors that might endanger the public or the environment;

As far back as the 12th century the Rabbi Maimonides wrote a Physicians oath with another line that I find appropriate:

May the love for my art actuate me at all time; may neither avarice nor miserliness, nor thirst for glory or for a great reputation engage my mind; for the enemies of truth and philanthropy could easily deceive me and make me forgetful of my lofty aim of doing good

A Doctor’s specialized knowledge and training puts his actions and opinions under a special light with regard to ethics.  Typically computer professionals live by a ‘do no harm’ credo as well, but our actions were usually bounded by our working domain.  With open data and blogs we can now use our skills and training for the betterment of democracy or as a lever to move a personal agenda.

Now that I am ready to wield this data in the name of Democracy and Truthiness I’m headed over to listen to Tom Janofsky talk about his experiences with Cloud Computing on EC2.

After all, once this great Philly data is out I’m going to need that elastic processing power….