It's not ready yet ...

I first ran into this phrase while working on VIVO.  It's a frustrating phrase used to push back at
requests.

• "Let's make an open source repository."
• "Is your change up yet?"
• "You should submit that to the project for others to use."

I thought it was just people new to open source.  People scared of submitting their code for possible judgement by a wider community.  I come from a "commit early, commit often, submit every day" philosophy.  My first programming position was pair programming, so though it was a single source I got used to talking about my decisions usually as I typed.

Recently, I heard this phrase come up again by someone who loves open source and uses many open source tools.  They wouldn't submit a tool to the team repository because it "wasn't ready".  It already had many useful things, however the individual wanted more features before giving it to everyone to use.  They were looking for perfection.

Perfection is in the eye of the beholder.  To me if it works, then it's perfect.  Build a feature, a method, anything you should release it to your team. Someone will find it useful, even if you don't think it will be.

This all comes from my pessimistic view of code, "it's all crap."  A co-worker and great friend told me this was too pessimistic a view.  Perhaps a better phrase is "Code is like a plant you nurture and grow.  Continue to nurture it with care and it will grow and get better.  Freeze it in time, like cutting a rose from the bush, and eventually it will wither and die."

We see it in our work all the time, today's great change is tomorrow's headache.  What worked well in the past, needs refactored in the future to expand its capability.  This isn't a bad thing, it's a great thing.  It means that code is a nearly living thing, constantly evolving.  Code that doesn't change and refactor dies as people move to better code bases.

How do we change this culture?  We have to change it in ourselves first.   Send our code up for review early, accept all criticism equally.  Change how we feel about others code.  Submit bug reports to software we use everyday.  Encourage our teams to submit early and give them positive and negative feedback and ask them questions.

• "This is a great idea."
• "I think you should do this instead."
• "Why did you choose this method?"

After we change ourselves, perhaps we can change the system to accept "Nothing's perfect, it is good to be ok."

Computer Science Education

SIGCSE, aka ACM's Special Interest Group for Computer Science Education, was in Denver Colorado a few weeks ago and it was kind of fitting that even though I wasn't there, I heard about one of the best ideas in Computer Science Education I've ever heard.

Jim Baker, whom I know through +Nicholas Skaggs  and Alex Viggio, is an Adjunct Professor at the University of Colorado Boulder teaching a class in Programming Languages.  The class seems pretty standard, reading assignments, homework, programming projects, papers, etc.  However, the innovative thing is how all of that is structured around GitHub.

I've mentioned this over several posts, version control is a missing element in most computer science education.  It's necessary to understand its benefits and to be used to the way it affects your coding style.  It's dramatically shaped mine, I now try to code in segments so I can commit complete thoughts and I check my code before pushing to the remote server after breaking things for other users.  Bad habits are started early and the earlier developers learn to use a version control system the less bad habits they'll acquire.

Students in Jim's class are required to acquire a GitHub account and register their username with the class. The class uses the students Github accounts to create repositories for them and push homework assignments into the student's repositories.  It's an innovative way to release homework assignments as it allows the instructor to modify them as questions come in or issues are discovered.  It was always annoying when a class website would get updated with a change to an assignment, but you'd only learn  4 days later during the next class session.  This style changes the standard class communication structure of the students pinging the course, in this case the website, to the instructor pinging his/her students "pull down the changes to the assignment".  It's a valuable lesson to learn, pull from your remote repository often.  It's so valuable that +Philip Chase once created a cheat sheet of daily tasks for his developers that had "git pull" as item number 1.  It seemed they were incapable of pulling or pushing to the remote repository and eventually their lack of proper process caused some major malfunctions.

Projects are submitted to a standard "import, test, and evaluate" CSE system.  We had something very similar at UF to check for cheating.  However, in addition to pushing to this system the student must also submit a pull request to the class.  Pull requests offer allow for peer review, you can see a diff of your code repository vs the source repository and receive feedback from the upstream developer (in this case the instructor).  This process is used in most open source projects and is beginning to gain traction in the corporate world with code review systems like Gerrit.  Learning this process adds practical training with their otherwise theoretical class assignments.  It's also provides an extra incentive to code well in the form of extra credit.  Code that does the tasks better than the instructors or in a more innovative way have their pull requests accepted and recieve extra credit.  If a student has tests in their code they can also receive extra credit.

The class has a paper and presentation requirement.  The paper has to be written in MarkDown, a standard wiki code.  Companies are increasing their use of dynamic documentation with corporate wiki's.  It allows the documentation to change on the fly and maintain a revision history without attempting to share a document and intangle yourself with locking and unlocking.  Markdown is also just a step away from Latex, which academics use in creating articles.  The papers, after being submitted, are required to receive reviews from two classmates.  Over the students professional careers they will be required to review and be reviewed when writing documentation.  This unique approach to paper submission readies the students for either the academic world or the corporate world.

I hope that this approach grows.  I know GitHub attended SIGCSE and attempted to expand git as an educational tool.  I'm hoping someone convinces Jim to write a paper about it and document the successes and the failures.  Until then ...

Software - Tradecraft or Science

Talking with one of my good friends in software development the other day I stumbled upon the question?  

Developing software, is it a tradecraft or is it a science?

The question came up while talking about the skills of various people where we used to work.  We had all kinds, people with masters degrees in software engineering, others with a bachelors degree in computer science.  Some didn't have a degree in software at all; such as my friend whose degree was in music.

What was interesting to note here is that degree should have indicated skill, if we were a pure science.  "I have studied more complicated theories then you and thus I am superior."  However, some of our most accomplished employees were some of our worse.  People with masters degrees and multiple classes in databases came up with some of our worst database designs.  Duplicated information in tables that should have been in a lookup table, using MyISM on a table that should have had foreign keys, the list of tragedies (in my opinion) go on.  Now I'm not an advocate for one normal form over another.  Going all the way out to fourth normal form can sometimes be extreme and a good software engineer should balance all aspects when building a program.  Its this thinking that started the wheels turning; software development or engineering has elements of a tradecraft.

In the sense that it's a trade, time tells you when by the book isn't necessary.  As you work at your trade you develop things like toolkits and acceptable short cuts.  In software we come up with tools for bug analysis, diffing files, profiling performance, writing code.  There are at least a few hundred articles on the web devoted to a developers "toolkit".  Some talk about it like finding that perfect set of wood working tools, where your hand has practically reshaped the handle from use.

In a purely technical sense it's also an art form.  Good code is not just functionally correct, but it's elegant.  Sometimes its the way the code touches the processor, ram, and disc in the most minimal and efficient way possible.  Other times its the way it does large amounts of effort with minimal amount of logic.  I'm not talking about a Perl one liner but simple, easy to read, powerful to use code.

I think like civil engineering and architecture, software engineering hints at the blending of these worlds; tradecraft and science.  The purely analytically with the artistic.  A building built on math alone would be a pretty boring building.  Architects learn to make a building soar.  In the same vein a great software engineer can make a program sing.  Don't get me wrong, we need the science too.  Without it our programs would be like the failed library building where the architect forgot to account for the weight of the books.  Just don't forget the craftmanship.

SimCity

When you've played every iteration of a game, like I have with Zelda, Assassins Creed, or Bioshock you see features come and go as the developers try to make the perfect game.  I've played SimCity since Simcity 2000 on Chris Pearson's old 450Z MHz Pentium II in high school.  I've seen the complexity get ramped up, and the size of the cities explode.  When I played the demo on Des' laptop for the latest SimCity, I was floored at how fun this game looked.  Sure the city was small, but services were simpler and the notification system was great.  We decided to buy two copies so we could play together and eagerly awaited its release.  Perhaps its just release day bugs, but I want my money back.

Stop making the web required

On the Meyer Briggs test I sit just barely over the line towards extrovert.  It's because in certain situations I'm a social butterfly; walking, talking, eating, etc.  However, in some situations I'm not; ie gaming. When I buy a game, I don't rush out to the multiplayer system.  I play single player all the way through and then if some of my friends are playing I'll try out the multiplayer.   I don't want to link to everyone all the time.  I shouldn't have to wait on a server to play my game.  

Many games are requiring online connections as a method of DRM.  If that's the way you want to play it, fine, require my connection at the first log in, verify my information, store my log in locally, and link me when I'm online.  If I have a laptop and want to play a game on the road or in a coffee shop, why am I getting denied because I don't have an active connection.  I can play the new StarCraft without a connection, as long as I had a connection when I installed it.  I won't earn rewards, but that's OK if all I'm looking for is a little fun.  

AutoSave does not replace Save

I don't know how many times I heard at the HelpDesk, "But I had autosave on!"  "Yes", I'd state, "lets check your settings, oh you had it saving every 8 hours, sorry you've lost your paper."  Now, nothing was more annoying in Final Fantasy than getting too into the game and forgetting to save.  Inevitably it was always when you had forgotten for an hour or two that suddenly the game froze or you were horribly killed.  However, you learned an important lesson every time, "save early and often".  

Desiree has exclaimed several times since SimCity has come out, "Crap it crashed" only to return and find her city 10 years younger than when she last saw it.  Just this past evening she was telling me about an arson problem in her SimCity and that she had finally gotten a police station to deal with it just before the crash.  Sure enough when the system let her sign in again, no police station.  

Let people save for themselves.  Let us manage our saves locally.  Its great that we can pick up wherever we left off on another computer, but don't take away the basics to give us this feature.  Simply warn us, "System saves do not transfer from computer to computer."

It's called a cluster

If the servers named North America East 1-3, West 1-2, Eastern Europe 1-2, Western Europe 1-2, etc are clustered servers, then I feel you need to send your engineers back to school.  Why in 2013 am I picking a server node.  I haven't picked a game server in years. Perhaps that's because I play on Xbox more often than anything else.  However, my last PC gaming experiences were League of Legends and StarCraft 2; both online games and I didn't pick the servers with either of them.  

Give me one huge node to hit and spread me and my friends traffic across the nodes with load balancing and clustering.  If you do it right you can simply add more nodes as traffic picks up ala Amazon. I guess that's the biggest problem.  OpenStack, Amazon EC2, and tools like Juju have shown us that clustering servers and spinning up new nodes can be trivial, if implemented correctly.

Tutorial, do I look like I ask for directions?

It's great that games come with tutorials.  If I ever have a problem, I often play through them.  However, if I've played the demo, or previous versions of the game I like to dive right in.  I'm currently stuck (one of the many reasons I'm writing and not playing) because the game says I need to play the tutorial, again (already played it once since it was released) and the tutorial keeps crashing.  

My favorite tutorials are in Call of Duty, go through door A for the tutorial or B to get into the action.  "Do you want the red pill or the blue pill?" Super easy to just jump right in or get a little help first.  

Tutorials are great.  

Good for you making one.  

Can I just play my game now?

Conclusion

It still looks like a fun game, and despite Desiree's frustrations she appears to be having fun.  However for me the game has been out for roughly 48 hours and I've played exactly 1 hour and attempted to play 2 additional hours.  That's the biggest flaw in this game.  If this was my first experience with SimCity, I'd never pick it up again.

Code Academy - Interesting Concept

Alex Viggio, my development lead at CU, sent me a link to the 2012 Crunchies Awards on Tech Crunch.  The winners were some of the usual suspects, but I saw a site that peaked my interest, CodeAcademy.com.  

Thier slogan is "Teaching to world to code." Now unlike a certain mayor, I don't believe everyone in the world can code.  It takes a 'different' mind than what everyone has.  I also don't believe everyone can paint, do carpentry, manage a team, fly a plane.  We're all different and we will find things beyond our abilities.  However, just because everyone won't be successful doesn't mean everyone shouldn't try.

I decided to take CodeAcademy for a spin.  It's important to me that people be taught to code in a responsible manner.  Bad habits develop early and lord knows I have my fair share.

The Profile

This is a requirement of almost any site, especially on the 2.0 web.  Unlike a lot of sites that attempt to integrate themselves with every known social site and application, CodeAcademy does a good job of selecting sites that represent developers

• Github - as a developer if you haven't at least tried git and github then I believe you either have A) been living under a rock for the last few years or B) are my father and work on mainframes.  This is a great place to send new developers.  The first question I usually ask during a interview is "How do you share code?"  I then usually face-palm as they say, "thumb drive" or "email".
• LinkedIn - lets face it facebook is for family and friends, LinkedIn is for co-workers and employers.  If you want to develop code professionally, you should have a LinkedIn account.
• Twitter - I know several developers use it to point out interesting coding articles they find.  It's not as important as the first two, but it has been useful in the past.

They have taken a page out of gaming that I love in web 2.0 sites, badges.  You see them all over StackExchange, Ubuntu, Fitocracy, and others.  It's like merit badges for adults, giving a little 'props' for completing something.  It's also a nice way to measure yourself against your peers and see areas where you can explore.  Perhaps this is just turning us into a needy praise driven zombies, but that's a more cynical observation of the reward system.  I perfer to look at anything that motivates us as a positive.

The rest of your profile reflects you current progress.  It shows the tracks you are currently working on and the courses you've completed.  It's a nice portal into your history allowing you to return to where you last left off.

Courses

When you first enter the "Learn" side of the the codeacademy interface you'll see a variety of tracks.  Some are coding languages: Python, Javascript, Ruby.  Others are more applied: 'API's, Web Fundamentals, Projects.  I think this is a great start, I just wish that the system had more formal languages,  C, C++, Java, and even some data languages and syntax, SQL, XPath, SPARQL.  All of the languages they presented are dynamically typed languages and I assume have a lighter interpreter since they are evaluated at run time instead of compile time.

Python Course

The first track I started was the Python course.  I already know a bit of python from some google courses and my own messing around, but its not a language I use every day.  I figured a bit of a refresher wouldn't be bad and would help me gauge the course since it's the closest to a newbie that I could get.

The courses starts off slow, going over the basics pretty much in order of any standard language book or first year programming course.  Syntax, basic data structures, conditional statements, looping, advanced data structures; its all laid out.   There are a few courses in the middle which re-iterate things like looping and feel a bit redundant. However, it's a good over view of the language, with a bit of programming basics thrown in for those without computer science degrees.

An interesting element of the track is that each course in the python track is paired with a project.  The projects attempt to apply a scenario to the information you learned in the course.  Some of them are quite well done, and allow you to define everything.  Other's are quite rigid and basically offer a paint by numbers approach.

The only issue I found with this track is that's several sections are quite buggy and do not offer assistance about why you don't pass.  For example, the challenge "Exam Statistics", requires you to computer the variance of a set of grades.  In python it looks something like 

grades = [100, 100, 90, 40, 80, 100, 85, 70, 90, 65, 90, 85, 50.5]
def print_grades(grades):
    for grade in grades:
        print grade
def grades_sum(grades):
    total = 0
    for grade in grades:
        total += grade
    return total
 
def grades_average(grades):
    sum_of_grades = grades_sum(grades)
    average = sum_of_grades / len(grades)
    return average
def grades_variance(grades, average):
    variance = 0
    for grade in grades:
        variance += (grade-average)**2
    variance = variance/len(grades)
    return variance

Which computes the answer 334.071005917. This is correct when used in the next section.  However, the section tells indicates something is wrong, "Oops try again."  That's the frustrating thing about some courses in code academy.  The feedback and review on the course is determined by the administrator.  Where one course will say "Oh you didn't print the string but the array. Print the string instead" another will just say "Oops" and hope you understand enough about why you failed.  The "Oops" courses seem to have the most restrictive requirements for passing each section and so it only increases the frustration.

HTML Course

I started this course because it was something I knew very well.  The speed of the course was very slow for me, but probably the right pace for my mother.  Its slow, re-iterates itself, and goes through every major structure of a webpage.  Overall if you know the basics of HTML, skip this course, unless you are curious like I was.

Despite the slow and boring nature of the course, I did appreciate the design that went into building it's interface.  Unlike the Python course which had a run environment at the bottom and code at the top, the HTML courses had tabs for various files (such as the index and css files here) and a panel to the right which shows the result of the code on the page.  It has a fairly active autosave, but includes a submit button to force display and call the course evaluation procedures.  It was clean easy to follow along and work in the panels.

Conclusion

While coding isn't for everyone, CodeAcademy does make it accessible to anyone willing to try.  It's courses are well thought out, and it appears to be actively expanding.  You'll find a few bumps in the road with courses that are too restrictive in their completion criteria or not descriptive enough in the requirements to pass.  However, the forums and bug tracker are active and responsive so hopefully time will fix those courses. 

It's not a bad resource for active developers that haven't touched a language or area of development before.  It also has a "Teach" section that allows developers to give back to CodeAcademy and share their knowledge with the community.  

For my part, I may try my hand at the teach section. I'll also keep an eye out for any new courses or tracks.

I leave you with a little poem from the first section in Python (which is from the Python library)

Zen of Python
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Appendix: Some sites about coding

• codeacademy.com
• code.org
• codinghorror.com

Three Tiered VIVO Build

An interesting aspect in the relationship between VIVO and Vitro (the non-ontology specific engine for VIVO and other projects) is the layered nature of the projects.  VIVO as a source repository is basically a shell that copies over it's Vitro base.

Cornell extends this further with adding their own changes to VIVO as a third level.  By modifying the build.xml file and the deploy.properties to include pointing at VIVO as a second layer, the build script can preform the same changes to VIVO as VIVO does to Vitro.

It seems a little complicated, but it removes you from the VIVO source allowing you to replace your VIVO target with the latest version and adapt more rapidly to new releases of VIVO.  There hasn't been a study yet, but I would guess that the average time from release of a new VIVO version to an institution upgrading to that version is probably around 5 months.  I know at UF it took us about that long (often waiting for the .1 release) and at CU they were still on 1.2 when I arrived (1.5 was released just before the VIVO conference in August).

I've started a wiki article on the new VIVO confluence wiki, https://wiki.duraspace.org/display/VIVO/Building+VIVO+in+3+tiers, that describes how to setup your local VIVO to run this three tier system.

VIVO Ingest in 1.5.1

Little update, about 3 months ago we moved from Gainesville, Florida to Boulder, Colorado as I accepted a position with the University of Colorado Boulder.  I'm back in the trenches, writing code everyday and working on VIVO for Faculty Affairs at CU Boulder.  Hopefully this will restart my attempts at writing more often and focus that writing on more technical and less managerial matters.

Updating Data Through Data Ingest in VIVO 1.5.1

I've had a lot of tasks at CU since starting, and I'll go over some of the things I've learned and written soon enough.  For now I wanted to talk about updating your Data in VIVO 1.5.1.  In the old days, when four programmers at UF embarked on Data Ingest, if you wanted to get data into VIVO you had to add it to the systems "Main Model" known as KB2.  This made data ingest difficult during the update phase because you either had to 

1. Start over from scratch with a blank VIVO
2. remove the previous data that you ingested which contained the data you want to change

Semantic Triple Stores, didn't have a key that we could use to link a row in KB2 to the data coming in from our source (in hindsight I believe there are ways with hash keys that we should have probably done this).   Due to this we constructed a very complicated (and time expensive) process to compare the data you are putting into VIVO against the last time you ingest from that source.  It creates an additions and a subtractions file which you then apply against the KB2 model.  Basically it was a bit like writing a remove and insert to accomplish an update in SQL.

Now in VIVO 1.5.1 this is mostly the same.  However, data doesn't have to be in the main model to be index.  So now we can separate our data by the source, or in the case of CU the tables that generated the data.  This allows for a shorter ingest process, we're only interested in dropping and adding to the models that have changes to be made.  I took CU's current process (which uses selenium scripts against the UI) and ripped out the portion that loaded data into KB2 (download of the data, use add/remove screen to load the exported data to the main graph) and I added a method to drop the import graph that we used.  This dropped my time from 3-4 hours to 1 hour for an entire ingest.

We were still rebuilding from scratch with each ingest and now that we're heading to production I wanted to make this process a little faster.  So I wrote the first of a couple of scripts towards automating the entire process.  This first script reviews the dat files for changes, allowing me to drop only the graphs that have changes and re-run thier ingest scripts.  

The process was fairly simply and I've included a couple of sites and blogs I used to figure out what to do.  By hand (which will become another script soon) I copied down the data from the previous run and ran a new export and copied down the new data.  I then pass to my new little script the two paths for the old data and the new data.  

The first step in the script is to review all of the file names and see what is missing, what's new.  Since the export process is an sql script which always uses the same file names for each of it's methods we don't have to worry misspellings, just new dat files or the lost of a dat file.

def reviewFolderForChanges(oldFilePath, newFilePath):

    #read in file path

    oldFiles = os.listdir(oldFilePath)

    newFiles = os.listdir(newFilePath)

    

    #get names of files in order of name by create list

    for newFile in newFiles:

        if newFile in oldFiles:

            reviewFileForChanges(oldFilePath+newFile, newFilePath+newFile)

        else:

            print "New File Found: " + newFile

            

    for oldFile in oldFiles:

        if oldFile not in newFiles:

            print "File Missing: " + oldFile 

 The next and final step in the process was to review the files themselves.  I could have used the python difflib, but I wanted more information from the files.  I wanted to know what was new and what had been removed.  Plus I found a nice little reference post by a Frankie Bagnardi that I wanted to implement myself.  The result has greatly increased the ability of the ingest operator (me today, probably alex and vance at other times) to make sure that the changes coming in are reflected in VIVO.  For example


Changes in file:/Users/stwi5210/Source/uccs-new-data/fis_faculty_member_positions.dat
- "http://vivo.uccs.edu/fisid_XXXXX","http://vivo.uccs.edu/deptid_XXXX","1435","Chair","http://vivoweb.org/ontology/core#FacultyAdministrativePosition","2","2"
- "http://vivo.uccs.edu/fisid_XXXXX","http://vivo.uccs.edu/deptid_XXXX","1419","Lecturer","http://vivoweb.org/ontology/core#FacultyPosition","5","4"


With information like this I go to the two individuals listed and make sure that they no longer have the positions of Chair or Lecturer.  This let me know that my ingest was successful and to report that it's ready to migrate to production.

All in all the script took about an hour to construct and then run and saved me about 40 minutes of ingesting.  Plus I was able to review VIVO after the ingests were finished for the data that should have changed, which is a big improvement over our previous methods of review.

Citation(s):

• Compare Two Files with Python by Frankie Bagnardi- http://aboutscript.com/blog/posts/107
• Python: iterate (and read) all files in a directory (folder) by Bogdon T - http://bogdan.org.ua/2007/08/12/python-iterate-and-read-all-files-in-a-directory-folder.html

Ubuntu Global Jam - Lessons I Learned

Ubuntu Cupcakes from Stacy Skaggs

The Ubuntu Global Jam was last Friday through Sunday and on Friday some Florida LoCo members got together at Chris Johnston's house in Orlando.  I've never been so I thought I would go, though it was on a Friday I could use the break.

I learned quite a few little things to make life easier working on Ubuntu Websites.  The first is a way of keeping the same development environments across project branches.  This has always been an issue for me because with bzr you can't just bzr checkout branch-name and the code changes like git.  Thats not its workflow.  Michael Hall pointed me to this post which is the method he uses for all his bzr repositories.  Basically, create a blank repository.  Inside the repository you create a trunk with the code from launchpad.  Then you branch the local copy into your feature/bug/issue branch.  Finally you lightweight checkout that branch to a working repo folder.  Sound confusing?  It does at first.  First off lightweight checkout doesn't import the code, just links to it.  Meaning when you modify code, push code, commit code its happening from the named branch not from work.  When you create your development environment around work, those files reference the work directory.  So when you change to a different feature, bug or issue the environment files still point to work and you don't have to rebuild.  Example commands:

bzr init-repo project-name
cd project-name
bzr branch lp:launchpad-bzr-name trunk
bzr branch trunk issue-specific-name
bzr checkout --lightweight issue-specific-name work

The second thing I learned was virtualenv for python.

sudo apt-get install python-virtualenv 
sudo apt-get install virtualenvwrapper

These separate the required environment for the ubuntu community websites from the installation of python you have installed and keeps everything nice and clean.  Now little did I know I was using this sort of with the make commands, however, with knowledge comes control and understanding.

I also got to see a bit of python dbugging as Michael Hall and Chris Johnston worked on that for a bit. Ate some fantastic cupcakes, and had a great time getting to dedicate some time to Ubuntu with some pretty awesome people.

Whose to blame for the misunderstanding ...

The incident I refer to is one happening more and more at work (and not just for me).  Examine the scenario:

A junior engineer has been given a feature to develop.  A senior engineer has built the basic mockup of the page, hooked it into the framework, but as the engineer attempts he runs into issues.  He asks, through email, how do I save to the database.  Its a project from a different development group, but the senior engineer has been through their code so he suggest looking into one part of the site and view specific controller and view files.  The junior engineer acknowledged and life moves on.  A week later the junior engineer comes to the senior engineer and says he is still having issues, so they open his code.  The junior engineer has created a new file (instead of following editing the files already apart of the application structure) and is attempting to drive logic that belongs in the controller in a file in the view.  He has ignored the structure of the site and is basically fighting against the natural flow of the program.  At first glance I'd say this is a lack of understanding of MVC.  However, this is not the first MVC project for this junior engineer.

Its a story playing itself over and over again in different ways.  At first glance, I'd assume the engineer never looked at the example files and project flow, instead they just looked up "PHP connect to database".  His code suggest that perhaps he looked at it, but didn't understand how the application uses the files.  Perhaps the senior engineer should have called the junior engineer over and shown them the files and explained the various methods in play to save and how they work with the view.

A friend of mine once mentioned a similar issue with contractors in the corporate landscape.  When he worked with them, over time he found himself practically programming the feature in the documentation to get the functionality he needed.  Inevitably they fought the existing application and built extraneous files.  At one point thier source had 4 different time classes that returned the time.

So how do we fix this without resorting to writing the program on a piece of paper and turning our junior engineers into a programming steno pool?  Is this a hiring issue, should we let junior engineers like this go?  A communication issue, did we not explain the solution completely should we have diagrammed it out?  A language barrier?  A training problem?  An experience issue?

At this time ... I'm lost, perhaps time will bring insight.

Passion Breeds Collaboration While Lack of it Breeds Indifference

I'll put the user in a box

Ever go to the doctor's office full of questions and ideas about a long term illness you may have?  Perhaps that's only me.

How did the doctor act?  Did he engage you in conversation?  Did he seem to ignore you?  Did he dismiss your questions?  Did he answer everything you asked and perhaps ask some of his own?

Have you ever been apart of an open source project?  On a mail list with other eager users and developers?  How did you react the last time someone sent in a question or suggestion?  Did you give them feedback?  Did you bypass the email based on subject?  Did you request more from them, perhaps code or testing?  Did you taunt them with your superior knowledge?

A friend of mine and I were talking about doctors and we came to the conclusion that the best doctors were the ones that wanted your input on your care.  It felt like it was because they were passionate about their jobs.  It seemed to us that the doctors who weren't either had lost the spark, or were only interested in medicine for the prestige, money, or perhaps family history (my father and his father and his father's father, etc).  That somehow led us to talking about open source projects and our time on one.  I came to realize that my passion for the project and for my career affected my involvement with the project.

When I was a developer I loved to hear from everyone.  I read every email, I was always on IRC.  As I moved into a manager role I became more remote.  I was hardly on IRC, I usually ignored listserv email.  Tickets from users were no longer wonderful feedback but annoying things that they could have fixed themselves.  My attitude dropped as I moved from that which I'm passionate about to something that I'm not.  So, I've stopped managing that project and tried to go back to engaging and coding.  The passion, its coming back, a bit slowly but only time will tell.

So the moral of this story?  When you are dealing with some a-hole professional don't be quick to judge.  Perhaps he made a wrong turn and is in a position that he has no passion for.  Instead of cursing him, say a little pray hoping he'll find his way to a profession he is passionate about. 

Engineer, Developer, Programmer, Analyst ... pick one?

I remember a while back my old boss mentioning he had chosen the working title of "Software Engineer" for positions at work because Engineers are not "web masters".  When I started this article I had been looking at positions on-line and I noticed a mix of definitions for the same work: Programmers, Software Developer, Software Engineer, Analyst.

From Dictonary.com

• Software Engineer - a person who designs and writes and tests computer programs 
• Software Developer - No Definition
• Analyst - a person who analyzes 
• Programmer - a person who writes computer programs.

Researching this post has lead me to the following conclusion.  A Software Engineer and a Software Developer are modern terminology for Analysts and a Programmers as Software as a field gets older and understands itself better.

Long ago we compartmentalized the field.  Analysts analyzed the needs of the customer and came up with the design then handed it off to programmers who actually created the application.  As the field has matured we learned a little bit about ourselves, we all like to design but secretly need to create too.  Software Engineers and Developers have come to embody a unified process of design, development, and implementation.  As you go up the technical ladder you design and mentor more than you write and test, but those things do not go away absolutely.

What is the difference between engineers and developers?  At first glance an engineer is a highly exact individual from a school of engineering.  However, in reality the difference is field of work.  Web companies, business groups, and application centered hardware agnostic positions tend to be described as Software Developers.  Technology companies and hardware centered positions tend to be described as Software Engineers.  As members of the software field both have to design the application to the users specification, write the application, and develop tests of that system.  Its the nature of the field.

Continuing Education for a Programmer

CSE at UF

One of my junior engineers asked the other day for resources to continue building up his experience with web development over the Christmas break.  So I put together a length email detailing items that I thought would be good for my team to read and do.  After re-reading the email, I realized that this was perfect blog material, so here it is un-cut from my original email.

Web Programming
The piece that makes web programming different from say standard gui applications is the web itself.  You have HTML and CSS and javascript on top of frameworks and server languages and databases.  Its a lot of different angles to become proficient in.  Start small and work your way in.  Some resources for learning more about the web.

• CSS Zen Garden - Why do we use CSS?  This is why, a perfectly structured site that can look completely different with the change in a style sheet.
• w3 schools - The W3 decides web standards, though browsers are sometimes lacking in following them.  They have a ton of tutorials on their site, check it out.

Computer Science Reading
Becoming better programmers doesn't always involve reading books about programming.  It could be a book about a developer, or a book about a methodology.  Some books you all should consider reading over your break

• The Cathedral and the Baazar - Ever heard of Eric S Raymond?  This essay talks about the differences between Open Source and traditional development.  The book contains his original essay and other articles he has written, you can find all of his articles on his site.
  
• The Art of Unix Programming - Another ESR Book about the philosophies of decisions made in building unix.  It can get preachy at times, but push through you'll be glad you did.
• The Mythical Man Month - Want to be a project or technical manager some day?  Read this book.  Its probably been suggested in class and if you were like me, you ignored it as you had three other classes suggesting 6 books a piece just like it.
  
• Just for Fun - The autobiography of Linus Torvolds.  It is not a technical book at all, but a very good read.
  
• The Design of Design - another Fred Brooks book, this one I have yet to start but it has excellent reviews and reading the overview it sounded like a good read for anyone in Software Development.
  
• Design Patterns - Very interesting book for those who love algorithm and technical design.  If you've taken the Oo class its required reading.

You should also consider adding to your weekly "programming/developer fitness program" reading blogs by programmers.  A few on my reading list are

• Joel on Software - Joel Spolsky has worked all over the map in software development, from programmer to manager to tech writer.  He isn't as prolific anymore, but there are some really great articles in his archive.  He is one half of the minds behind Stack Overflow and created Trello
  
• Coding Horror - Jeff Atwood another almost former blogger he has been picking up momentum again recently.  He is the other half behind Stack Overflow.
• Mark Shuttleworth  - Started following his blog after last years UDS, he has some interesting posts on a variety of subjects.
  

Those are the individuals I follow currently, though my actual list is quite larger I've stopped reading some of my .Net blogs since joining CTRIP.  Another reading strategy is to follow planets that aggregate blogs.  A few planets to think about

• Ubuntu
• Gnome
• Java.org
• Python
• Perl
  
• Apache
• Linux Kernal

If you have been around technology for a while you may already have a couple of technology sites that you read ever day.  For me its

• The Linux Journal  - the original linux magazine
  
• LWN - from NickS its another source of linux news
  
• Ars Technica - A great aggregator of technical and science news
  
• Wired - Its wired the magazine on the web
  
• Slashdot - another technology news website

"Google" New Ideas in Software Development
I keep throwing out things like Code Review, Gerrit, and Jenkins during our standups.  These are things that are coming down the pipeline at CTRIP, but why are they coming at all?  Being a better developer involves immersing yourself in new technologies and methodologies and being able to distinguish the fads from the game changers.  Use Google and Wikipedia to your advantage and look up things you are hearing in class and around the office.  Some key terms to "google" that have been floating around lately

• Agile Development
• KanBan
• Bazaar
• Gerrit
• Code Review
• Testing (there are different kinds than just Unit, Integration, and Acceptance)
• Jenkins
• HTML 5
• SPARQL
• Semantic Web
• Linked Open Data
• Bio-Medical Informatics vs BioInformatics
• Clinical Trial
• Personalized Medicine
• Translational Science

Try other frameworks!
You may notice that I often give you all(my team as this was originally written for just them) a vague idea of how it should work and then expect you to go find the actual solution.  Why is that?  Its because I have experience in other frameworks and know how to do what we are talking about in them, but perhaps not exactly in grails.  Learning additional web frameworks and in other languages will often connect concepts in your primary framework that you haven't already grasped.  Think of it as looking at web programming from a different angle.  The perspective is different so there are differences, but their are also similarities.  A couple of good frameworks to get started in:

• CakePHP (this is our other framework at CTRIP/ESE)
• Django - a python framework similar to cake and grails
• ASP.NET MVC 3 - a complete 180 from our usual style but not a bad idea to understand a completely different viewpoint.  As students you should be able to get VS 2008 for free or use Visual Web Developer Express.  Use C# as its closest to Java.  Just be careful of the differences like toString() which in C# is ToString() ... very annoying.
  
• Spring - one of the layers that Grails is built upon.  Its a good idea to learn Spring without Grails to better understand Grails itself
  

Learning is often doing, you can't just read it from a book and expect to be able to do it.  Find tutorials in these frameworks and take them for a test drive.  You'll learn a lot about web development in the process.

IDE's: is there a tool for every occasion?

IDE, Integrated Development Environment, a programmer's best friend.  The best ones run unit tests, pre-compile code, do code completion, and run the code in a demo environment.

The value of a second pair of eyes

A long time a go in an office far far away I had to sit for four hours a day watching someone else write code.  It was slow, it was boring, and it drove me nuts.  It was pair programming and in retrospect I've noticed that it did make both of us better programmers and the code itself better.  I often get little reminders of one of the important reasons for pair programming.

A second set of eyes

This weeks example, I received an IRC message asking if I could help out with a CSS issue.  I'm not a "CSS guy" I'm more a "CSS thief" but I was game.  So, we looked over his Django template and a solution eventually presented itself.  I mocked it up, uploaded a patch and he merged it in.  Great!  Now he states if he could only figure out one other little issue.  

I couldn't help myself, I was in programming mode.  I asked "what was wrong?"  A ManyToMany object wasn't printing out on his page and had him stumped.  It stumped me to, I barely know Django and after going through all the documentation everything looked correct.  I threw in a "{{ forloop.counter }}" and nothing showed up.  Found sites that were doing exactly what he was trying to do and they matched.  Finally I saw he had one little element "related_name" in the foriegnkey attribute.  I had read about it, and it wasn't necessary so I thought "why not try removing it and seeing what happens."  His code worked.  Code he and then I had poured over for hours that was technically correct and one little "un-necessary" element was the cause.

Programming: Putting all of the Pieces together (Part 4)

While teaching the developers the proper process of coding and testing and committing will make them better developers, I need something to help me monitor the process.  So far all I have is labor intensive reviewing individual commits and running the tests by hand.  Pulling back in things learned from cornell while working on VIVO I can automate my tests with Jenkins.

Jenkins is amazing, I've spent part of the previous week putting together a POC installation on a Virtual Machine and the tests that I can run.  There are plugins for Selenium and Grails and code review like codenarq and code coverage.  With this we can automate tests to be performed before code is merged with develop.  If our developers are following the appropriate steps, their code will pass with flying colors.  If they are not following the appropriate steps, I save time and effort by having a robot (jenkins) remind them to test their code and that the code they are trying to submit fails.

How do I put this into our typical development process?  We're using in addition to Git, Git-Flow.  Basically it formalizes our branching structure so that between projects we are all following the same methods for defining the production branch (master) the development branch (develop) the working branches (/feature) and branches for hot fixes, version tags, etc.  Inserting jenkins into my development process isn't as easy as targeting a branch.  If I select master, I'm not testing often enough as its only receives release candidates from develop.  I could select develop, however, I want to insert it before the code comes into the develop.  I want to tigger it when my developer in his branch calls

git flow feature finish featureName

Also I don't want to only rely upon automated review.  What if my developer didn't add the appropriate tests for his or her new feature.  What if the poor code is syntactically perfect, but lacks good design decisions.  The code review tools will catch a lot of these errors, but not all of them.  Re-enter the original concept of code reviews.  When your peers see your code, you code cleaner, smarter, more efficiently.  Enter Gerrit, a code review system that automates the process of reviewing code.  It sets up review tickets, and can be integrated with jenkins to allow jenkins to weight in a pass/fail based on the scripted tests.  Reviewers can be setup for each developer and I can get emails when there is code for me to review.  Upon successful review the code can be automatically merged back into our main development trunk.

So this is the task this week; put it all together.  I have the proper pieces falling into place, now I need to make them match up.  How do I interrupt the git-flow process to insert Gerrit?  How do I make Gerrit preform the rest of the merge with develop if the reviewers and Jenkins approve?  

RIP Anne Mccaffrey

Before my mother began reading romance novels (yuck), she read fantasy books.  More specifically, she read Anne Mccaffrey.  My mother pulled from her collection my first fantasy book, Dragonsong.  It was a quick read, but it took me to a far away world.  The threat of thread, ancient lifestyle, dragons it all took hold.  I quickly moved from DragonSong to DragonSinger and DragonDummer finishing up the Harper Hall Trilogy.  From there the other Pern books which took on a slight scifi bent, then the "Ship Who Sang" and many other series by Mccaffrey.  My cousin suggested one of my other favorite authors, David Eddings, because I liked Anne Mccaffrey.

Anne Mccaffrey died this past week from a stroke.  Anne, you brought music into a genre of of swords, magic and mythical beasts.  Your books took me out of childrens literature and opened me to hundreds of authors and worlds.

Thank you

NPR Article on Anne Mccaffrey

Programming: Putting all of the Pieces together (Part 3)

Testing the individual units of the code is great for making sure each unit works.  Integration testing is fantastic for making sure the code fits together as intended.  What about testing the interface from the users prospective?

It all starts with plan, when I was a developer at the library we spent over a year combining 4 disparate ticket tracking and inventory systems into a single system that could link library hardware to the request for its purchase, any trouble tickets, etc.  As we were finishing the project our boss asked for a test document.  When complete it was a hundred and fourteen pages complete with step by step tests to be performed from the user interface.  Running through every page would take us about two to three hours.

A few years older and a few years smarter and I still make my team write those test plans.  However, today I'm making them create parallel selenium tests for every item in the document.  The current project is around 50 pages and the tests take 5 minutes, on slow.  The best part about this, selenium doesn't get bored and skip tests.  I don't know how many times, my own boredom caused me to miss an error in a function that I hadn't anticipated would be affected by my change.

So at this point I have tons of tests being generated.  I'm working on changing the workflow of my team so that they analyze the problem, start a new branch, build tests, code and comment, run tests, then push to our main branch.  They are starting to get better, but how do I take this from "you should do this" to "this happens every time". . .

Programming: Putting all of the Pieces together (Part 2)

Lets take a look at what I have to start with:

• Bug and Feature Tracking using Fusion Forge
• Git Repository
• Git Flow
• Grails Framework (Built on Groovy, Built on Spring, Built on Java)

What did I learn from VIVO:

• Style enforcement through eclipse
• Hudson/Jenkins for Continuous Integration
• Seleninum for Acceptance Testing

The tools are there, now its time to start putting them together.  Our old projects, our non-VIVO projects, have never used unit tests.  Its a change in style that even our more senior engineers are just now starting to do.  What is a unit test?  According to Wikipedia it is "a method of testing an individual unit of a software project."  In our case, it is a test of each class in our Grails project.  Its interesting teaching developers to test.  If a model (or domain) accepts null values, do you have to test that it will save and validate when given a null value?  Of course you do, but until that's a part of your programming knowledge base it might not occur to you.  If you test that a model properly fails validation when given a null value for a field that is null, do you have to test for the error that is returned?  Yes, just testing for the validation doesn't allow you to confirm that a the view will be given the proper error.

Lesson 1, when changing the culture of development at your group you may encounter slackers.

Lesson 2, be specific when you say "fix the tests", explain that means fixing either the code or the test depending on the reason the test is failing.

So, we are learning as a group to test more.  Unit tests lead to integration tests, which thankfully in Grails are built similar to unit tests.  We've stumbled, found tests that are basically integration tests in unit tests and vice versa.  Found items that should have been tested for and failed later.  Next step, acceptance tests!

Programming: Putting all of the Pieces together (Part 1)

It struck me the other day as I attempted to navigate my project's git log to review code that my team had pushed into the repository, something was missing.  It's taken a long time to pull myself out of the student programmer headspace and towards a full fledged developer whose code is sacred and must retain its history.

A Little History
I didn't get started with computers until around 2000.  Of course we had one at home, but my father is a programmer and for him at the end of a long day working for a bank he didn't want or need a computer at home.  Until 1994 we had a little black and white dos box.  That was replaced by a 386 with windows 3.1 until 1999 when I would get my own computer for college.

Fast forward through 5 years of school 2 spent as pre-med and 3 as computer science and enter my first programming job at the UF Computing Helpdesk.  It was a part time job with-in a part time job.  When items were finished they went to production and that was that.

From there I went to work at Smathers Libraries a Microsoft shop.  We used Visual Studio and Visual Source Safe at first.  For those who are rabid against MS, set aside your anger. Visual Studio is quite nice even though it is proprietary.  Visual Source Safe on the other hand, absolutely abysmal.  It would claim files were locked by one user only to find that person didn't have the file locked and could not unlock it.  It would over write newer files with older files when a person saved.  In summary, it stunk.

So we switched to the next best thing, in our uneducated opinions, versioned out file folders.  Basically a by hand version control system.  We were pair programming a rather large project so every morning the code would get copied to the file share and appended with the date and prefix AM.  In the afternoon the same would occur with the prefix PM.  It worked fairly well, the file share was backed up on tape and lived on a SAN that had a redundant system across campus.  Sure we had to navigate folders to find old versions of code, but we hardly ever looked back.  Who would need to do that?

Back to Today
Since VIVO, I've been immersed in SVN, Mercurial, CVS, Bazaar and Git.  I've learned the ins and outs of my old pal from undergrad, Eclipse.  I've used various bug and feature tracking systems; SourceForge, Jira, Launchpad, and FusionForge.  Its all coming together, the lifecycle that is software development.  The code is simply the result of all the pieces falling into place and without the right pieces, the code usually suffers.

Back to my current delima.  I have very junior programmers.  They come from a mixture of environments and educational levels, and for all of them I'm deeply concerned with what they took away from those experiences.  For example, perhaps its the sum of all of my experiences or perhaps its the first school project that nearly missed a deadline due to a hard coded value but I strive not to do these things.  I found, when helping a masters student fix a table filtering issue, hard coded drop down lists for items that are stored in the database.  The breakage was due to spelling differences between the database and the dropdownlist, obviously.  Their code is scary and the worst part I am putting my and the departments reputation behind it.

So how do I help them learn and avoid accepting bad code.  My brain slowly starts up, the hamster rising from his nap, and I think "Review their code".  So I open the code and before I begin to browse the first spark hits, use the gitg explorer so you can review what they are changing.  "Duh! How can I help them improve if I don't know who committed the error."  Now my hamster is not yet up to speed and when we finally got up to speed I processed that something was missing.  Why am I doing this code review by hand?  I have standards documents and with VIVO I've used code coverage reports and unit tests and acceptance tests.  How do I put this all together?

A Programmers Conundrum: Doing the same thing in a completely different way

I've begun to embark on a quest to understand development in Ubuntu and make myself a more active user.  This quest led me to getting involved with guys that develop the Loco Team Portal and Summit site.  Its python, django, sql, and bazaar.  Three of the four I have never used before.  I've done web development before.  I've used C#, Java, and PHP.  I've used Asp.NET, Grails, and CakePHP.  I've used Visual SourceSafe, Subversion, and Git.  On paper this is simple, similar task new frameworks for doing it.  Reality is much more difficult.