Continuing Education for a Programmer

CSE at UF

One of my junior engineers asked the other day for resources to continue building up his experience with web development over the Christmas break.  So I put together a length email detailing items that I thought would be good for my team to read and do.  After re-reading the email, I realized that this was perfect blog material, so here it is un-cut from my original email.

Web Programming
The piece that makes web programming different from say standard gui applications is the web itself.  You have HTML and CSS and javascript on top of frameworks and server languages and databases.  Its a lot of different angles to become proficient in.  Start small and work your way in.  Some resources for learning more about the web.

• CSS Zen Garden - Why do we use CSS?  This is why, a perfectly structured site that can look completely different with the change in a style sheet.
• w3 schools - The W3 decides web standards, though browsers are sometimes lacking in following them.  They have a ton of tutorials on their site, check it out.

Computer Science Reading
Becoming better programmers doesn't always involve reading books about programming.  It could be a book about a developer, or a book about a methodology.  Some books you all should consider reading over your break

• The Cathedral and the Baazar - Ever heard of Eric S Raymond?  This essay talks about the differences between Open Source and traditional development.  The book contains his original essay and other articles he has written, you can find all of his articles on his site.
  
• The Art of Unix Programming - Another ESR Book about the philosophies of decisions made in building unix.  It can get preachy at times, but push through you'll be glad you did.
• The Mythical Man Month - Want to be a project or technical manager some day?  Read this book.  Its probably been suggested in class and if you were like me, you ignored it as you had three other classes suggesting 6 books a piece just like it.
  
• Just for Fun - The autobiography of Linus Torvolds.  It is not a technical book at all, but a very good read.
  
• The Design of Design - another Fred Brooks book, this one I have yet to start but it has excellent reviews and reading the overview it sounded like a good read for anyone in Software Development.
  
• Design Patterns - Very interesting book for those who love algorithm and technical design.  If you've taken the Oo class its required reading.

You should also consider adding to your weekly "programming/developer fitness program" reading blogs by programmers.  A few on my reading list are

• Joel on Software - Joel Spolsky has worked all over the map in software development, from programmer to manager to tech writer.  He isn't as prolific anymore, but there are some really great articles in his archive.  He is one half of the minds behind Stack Overflow and created Trello
  
• Coding Horror - Jeff Atwood another almost former blogger he has been picking up momentum again recently.  He is the other half behind Stack Overflow.
• Mark Shuttleworth  - Started following his blog after last years UDS, he has some interesting posts on a variety of subjects.
  

Those are the individuals I follow currently, though my actual list is quite larger I've stopped reading some of my .Net blogs since joining CTRIP.  Another reading strategy is to follow planets that aggregate blogs.  A few planets to think about

• Ubuntu
• Gnome
• Java.org
• Python
• Perl
  
• Apache
• Linux Kernal

If you have been around technology for a while you may already have a couple of technology sites that you read ever day.  For me its

• The Linux Journal  - the original linux magazine
  
• LWN - from NickS its another source of linux news
  
• Ars Technica - A great aggregator of technical and science news
  
• Wired - Its wired the magazine on the web
  
• Slashdot - another technology news website

"Google" New Ideas in Software Development
I keep throwing out things like Code Review, Gerrit, and Jenkins during our standups.  These are things that are coming down the pipeline at CTRIP, but why are they coming at all?  Being a better developer involves immersing yourself in new technologies and methodologies and being able to distinguish the fads from the game changers.  Use Google and Wikipedia to your advantage and look up things you are hearing in class and around the office.  Some key terms to "google" that have been floating around lately

• Agile Development
• KanBan
• Bazaar
• Gerrit
• Code Review
• Testing (there are different kinds than just Unit, Integration, and Acceptance)
• Jenkins
• HTML 5
• SPARQL
• Semantic Web
• Linked Open Data
• Bio-Medical Informatics vs BioInformatics
• Clinical Trial
• Personalized Medicine
• Translational Science

Try other frameworks!
You may notice that I often give you all(my team as this was originally written for just them) a vague idea of how it should work and then expect you to go find the actual solution.  Why is that?  Its because I have experience in other frameworks and know how to do what we are talking about in them, but perhaps not exactly in grails.  Learning additional web frameworks and in other languages will often connect concepts in your primary framework that you haven't already grasped.  Think of it as looking at web programming from a different angle.  The perspective is different so there are differences, but their are also similarities.  A couple of good frameworks to get started in:

• CakePHP (this is our other framework at CTRIP/ESE)
• Django - a python framework similar to cake and grails
• ASP.NET MVC 3 - a complete 180 from our usual style but not a bad idea to understand a completely different viewpoint.  As students you should be able to get VS 2008 for free or use Visual Web Developer Express.  Use C# as its closest to Java.  Just be careful of the differences like toString() which in C# is ToString() ... very annoying.
  
• Spring - one of the layers that Grails is built upon.  Its a good idea to learn Spring without Grails to better understand Grails itself
  

Learning is often doing, you can't just read it from a book and expect to be able to do it.  Find tutorials in these frameworks and take them for a test drive.  You'll learn a lot about web development in the process.

IDE's: is there a tool for every occasion?

IDE, Integrated Development Environment, a programmer's best friend.  The best ones run unit tests, pre-compile code, do code completion, and run the code in a demo environment.

The value of a second pair of eyes

A long time a go in an office far far away I had to sit for four hours a day watching someone else write code.  It was slow, it was boring, and it drove me nuts.  It was pair programming and in retrospect I've noticed that it did make both of us better programmers and the code itself better.  I often get little reminders of one of the important reasons for pair programming.

A second set of eyes

This weeks example, I received an IRC message asking if I could help out with a CSS issue.  I'm not a "CSS guy" I'm more a "CSS thief" but I was game.  So, we looked over his Django template and a solution eventually presented itself.  I mocked it up, uploaded a patch and he merged it in.  Great!  Now he states if he could only figure out one other little issue.  

I couldn't help myself, I was in programming mode.  I asked "what was wrong?"  A ManyToMany object wasn't printing out on his page and had him stumped.  It stumped me to, I barely know Django and after going through all the documentation everything looked correct.  I threw in a "{{ forloop.counter }}" and nothing showed up.  Found sites that were doing exactly what he was trying to do and they matched.  Finally I saw he had one little element "related_name" in the foriegnkey attribute.  I had read about it, and it wasn't necessary so I thought "why not try removing it and seeing what happens."  His code worked.  Code he and then I had poured over for hours that was technically correct and one little "un-necessary" element was the cause.

Programming: Putting all of the Pieces together (Part 4)

While teaching the developers the proper process of coding and testing and committing will make them better developers, I need something to help me monitor the process.  So far all I have is labor intensive reviewing individual commits and running the tests by hand.  Pulling back in things learned from cornell while working on VIVO I can automate my tests with Jenkins.

Jenkins is amazing, I've spent part of the previous week putting together a POC installation on a Virtual Machine and the tests that I can run.  There are plugins for Selenium and Grails and code review like codenarq and code coverage.  With this we can automate tests to be performed before code is merged with develop.  If our developers are following the appropriate steps, their code will pass with flying colors.  If they are not following the appropriate steps, I save time and effort by having a robot (jenkins) remind them to test their code and that the code they are trying to submit fails.

How do I put this into our typical development process?  We're using in addition to Git, Git-Flow.  Basically it formalizes our branching structure so that between projects we are all following the same methods for defining the production branch (master) the development branch (develop) the working branches (/feature) and branches for hot fixes, version tags, etc.  Inserting jenkins into my development process isn't as easy as targeting a branch.  If I select master, I'm not testing often enough as its only receives release candidates from develop.  I could select develop, however, I want to insert it before the code comes into the develop.  I want to tigger it when my developer in his branch calls

git flow feature finish featureName

Also I don't want to only rely upon automated review.  What if my developer didn't add the appropriate tests for his or her new feature.  What if the poor code is syntactically perfect, but lacks good design decisions.  The code review tools will catch a lot of these errors, but not all of them.  Re-enter the original concept of code reviews.  When your peers see your code, you code cleaner, smarter, more efficiently.  Enter Gerrit, a code review system that automates the process of reviewing code.  It sets up review tickets, and can be integrated with jenkins to allow jenkins to weight in a pass/fail based on the scripted tests.  Reviewers can be setup for each developer and I can get emails when there is code for me to review.  Upon successful review the code can be automatically merged back into our main development trunk.

So this is the task this week; put it all together.  I have the proper pieces falling into place, now I need to make them match up.  How do I interrupt the git-flow process to insert Gerrit?  How do I make Gerrit preform the rest of the merge with develop if the reviewers and Jenkins approve?  

RIP Anne Mccaffrey

Before my mother began reading romance novels (yuck), she read fantasy books.  More specifically, she read Anne Mccaffrey.  My mother pulled from her collection my first fantasy book, Dragonsong.  It was a quick read, but it took me to a far away world.  The threat of thread, ancient lifestyle, dragons it all took hold.  I quickly moved from DragonSong to DragonSinger and DragonDummer finishing up the Harper Hall Trilogy.  From there the other Pern books which took on a slight scifi bent, then the "Ship Who Sang" and many other series by Mccaffrey.  My cousin suggested one of my other favorite authors, David Eddings, because I liked Anne Mccaffrey.

Anne Mccaffrey died this past week from a stroke.  Anne, you brought music into a genre of of swords, magic and mythical beasts.  Your books took me out of childrens literature and opened me to hundreds of authors and worlds.

Thank you

NPR Article on Anne Mccaffrey

Programming: Putting all of the Pieces together (Part 3)

Testing the individual units of the code is great for making sure each unit works.  Integration testing is fantastic for making sure the code fits together as intended.  What about testing the interface from the users prospective?

It all starts with plan, when I was a developer at the library we spent over a year combining 4 disparate ticket tracking and inventory systems into a single system that could link library hardware to the request for its purchase, any trouble tickets, etc.  As we were finishing the project our boss asked for a test document.  When complete it was a hundred and fourteen pages complete with step by step tests to be performed from the user interface.  Running through every page would take us about two to three hours.

A few years older and a few years smarter and I still make my team write those test plans.  However, today I'm making them create parallel selenium tests for every item in the document.  The current project is around 50 pages and the tests take 5 minutes, on slow.  The best part about this, selenium doesn't get bored and skip tests.  I don't know how many times, my own boredom caused me to miss an error in a function that I hadn't anticipated would be affected by my change.

So at this point I have tons of tests being generated.  I'm working on changing the workflow of my team so that they analyze the problem, start a new branch, build tests, code and comment, run tests, then push to our main branch.  They are starting to get better, but how do I take this from "you should do this" to "this happens every time". . .

Programming: Putting all of the Pieces together (Part 2)

Lets take a look at what I have to start with:

• Bug and Feature Tracking using Fusion Forge
• Git Repository
• Git Flow
• Grails Framework (Built on Groovy, Built on Spring, Built on Java)

What did I learn from VIVO:

• Style enforcement through eclipse
• Hudson/Jenkins for Continuous Integration
• Seleninum for Acceptance Testing

The tools are there, now its time to start putting them together.  Our old projects, our non-VIVO projects, have never used unit tests.  Its a change in style that even our more senior engineers are just now starting to do.  What is a unit test?  According to Wikipedia it is "a method of testing an individual unit of a software project."  In our case, it is a test of each class in our Grails project.  Its interesting teaching developers to test.  If a model (or domain) accepts null values, do you have to test that it will save and validate when given a null value?  Of course you do, but until that's a part of your programming knowledge base it might not occur to you.  If you test that a model properly fails validation when given a null value for a field that is null, do you have to test for the error that is returned?  Yes, just testing for the validation doesn't allow you to confirm that a the view will be given the proper error.

Lesson 1, when changing the culture of development at your group you may encounter slackers.

Lesson 2, be specific when you say "fix the tests", explain that means fixing either the code or the test depending on the reason the test is failing.

So, we are learning as a group to test more.  Unit tests lead to integration tests, which thankfully in Grails are built similar to unit tests.  We've stumbled, found tests that are basically integration tests in unit tests and vice versa.  Found items that should have been tested for and failed later.  Next step, acceptance tests!

Programming: Putting all of the Pieces together (Part 1)

It struck me the other day as I attempted to navigate my project's git log to review code that my team had pushed into the repository, something was missing.  It's taken a long time to pull myself out of the student programmer headspace and towards a full fledged developer whose code is sacred and must retain its history.

A Little History
I didn't get started with computers until around 2000.  Of course we had one at home, but my father is a programmer and for him at the end of a long day working for a bank he didn't want or need a computer at home.  Until 1994 we had a little black and white dos box.  That was replaced by a 386 with windows 3.1 until 1999 when I would get my own computer for college.

Fast forward through 5 years of school 2 spent as pre-med and 3 as computer science and enter my first programming job at the UF Computing Helpdesk.  It was a part time job with-in a part time job.  When items were finished they went to production and that was that.

From there I went to work at Smathers Libraries a Microsoft shop.  We used Visual Studio and Visual Source Safe at first.  For those who are rabid against MS, set aside your anger. Visual Studio is quite nice even though it is proprietary.  Visual Source Safe on the other hand, absolutely abysmal.  It would claim files were locked by one user only to find that person didn't have the file locked and could not unlock it.  It would over write newer files with older files when a person saved.  In summary, it stunk.

So we switched to the next best thing, in our uneducated opinions, versioned out file folders.  Basically a by hand version control system.  We were pair programming a rather large project so every morning the code would get copied to the file share and appended with the date and prefix AM.  In the afternoon the same would occur with the prefix PM.  It worked fairly well, the file share was backed up on tape and lived on a SAN that had a redundant system across campus.  Sure we had to navigate folders to find old versions of code, but we hardly ever looked back.  Who would need to do that?

Back to Today
Since VIVO, I've been immersed in SVN, Mercurial, CVS, Bazaar and Git.  I've learned the ins and outs of my old pal from undergrad, Eclipse.  I've used various bug and feature tracking systems; SourceForge, Jira, Launchpad, and FusionForge.  Its all coming together, the lifecycle that is software development.  The code is simply the result of all the pieces falling into place and without the right pieces, the code usually suffers.

Back to my current delima.  I have very junior programmers.  They come from a mixture of environments and educational levels, and for all of them I'm deeply concerned with what they took away from those experiences.  For example, perhaps its the sum of all of my experiences or perhaps its the first school project that nearly missed a deadline due to a hard coded value but I strive not to do these things.  I found, when helping a masters student fix a table filtering issue, hard coded drop down lists for items that are stored in the database.  The breakage was due to spelling differences between the database and the dropdownlist, obviously.  Their code is scary and the worst part I am putting my and the departments reputation behind it.

So how do I help them learn and avoid accepting bad code.  My brain slowly starts up, the hamster rising from his nap, and I think "Review their code".  So I open the code and before I begin to browse the first spark hits, use the gitg explorer so you can review what they are changing.  "Duh! How can I help them improve if I don't know who committed the error."  Now my hamster is not yet up to speed and when we finally got up to speed I processed that something was missing.  Why am I doing this code review by hand?  I have standards documents and with VIVO I've used code coverage reports and unit tests and acceptance tests.  How do I put this all together?

A Programmers Conundrum: Doing the same thing in a completely different way

I've begun to embark on a quest to understand development in Ubuntu and make myself a more active user.  This quest led me to getting involved with guys that develop the Loco Team Portal and Summit site.  Its python, django, sql, and bazaar.  Three of the four I have never used before.  I've done web development before.  I've used C#, Java, and PHP.  I've used Asp.NET, Grails, and CakePHP.  I've used Visual SourceSafe, Subversion, and Git.  On paper this is simple, similar task new frameworks for doing it.  Reality is much more difficult.

Ubuntu Development Tutorial

A while back I was trying to remember everything I needed to setup for developing things on Ubuntu.  How do I set up bzr?  What keys do I need to setup on the machine?  Then a friend of mine sent me a link to several articles on omgubuntu.co.uk.  They are the perfect guide for getting started or for just remembering everything you need to setup when moving machines.  For that second reason I'm putting all of them up here.

• Part 1
• Part 2
• Part 3
• Part 4
• Part 5
• Part 6

The articles were written by Daniel Holbach at OMG Ubuntu.  If you are ever considering developing on ubuntu or just wanting to learn more about its development I suggest you check them out.

The Ubuntu Developer Summit (UDS-P)

Precise Pangolin Sign borrowed from tipforlinux.com

I attended the Ubuntu Developer Summit last week in Orlando Florida.  While its my second UDS, the first I attended more as a guest, an outside observer who used Ubuntu and was developing an open source application.  Luck has it that UDS was held in Orlando one more time and with this UDS a chance to finally get involved.

Now a co-worker and I both decided to go and at first we figured we'd drive down and back every day.  The theory was its 10 dollars cheaper than the cheapest hotel and we'd get to sleep in our own beds.  By Wednesday that was out the window and we stayed at the hotel next door (60 dollars cheaper a night and right next door).

Day 1

The opening keynote went well, Jono stood up and talked for a bit about UDS and then Mark came up and talked about Ubuntu and someone from Linaro (sorry it was a week ago and I forget his name) talked about Linaro's progress.  Mark dropped the plans for 2014, Ubuntu on everything (phone's, tablets, TV's, computers, etc).  The most amazing thing about this plan isn't that its Ubuntu running on each of those devices, but his second image had Ubuntu running on a phone driving the other devices.  Kind of what Motorola has faked in my phone where I plug into a monitor and suddenly my dual core 1GHZ, 1GB of RAM cell phone is a desktop PC.

Morning Sessions

I first attended the Community Round Table. Jono, or whoever thought it up, had a great plan to hold round tables every morning. I didn't get to attend all of them, but they set the stage for the rest of the day. In this first round table, several community sessions were added to the weekly scheduled based on items from the round table. Next up was the Software Center Roadmap. I barely remember a thing, I need to write these closer to the actual event next time.

Afternoon Session

In the afternoon, I went to Unity Desktop Integration and Ubuntu and Tablets. Ubuntu and Tablets was the most interesting. The discussion became very detail driven. The kernel needs to be smaller, faster, less of memory hog, less of a power hog. Several topics brought up Android as our Java Desktop Linux cousin. There was some discussion of the user interface that unity had to change. Talk of tile menus vs the current layout. My hope is that with 2 years we can do something better than duplicate the tile system of android and ios. Unity to me, seems like it would work. The top menu bar would need to be something else, like a button driven menu ala android (I know I'm already picking apart the others). I didn't get to point out my webdesign thought for the desktop. Think css zen garden. Ubuntu would detect the screen size and apply the "css" style for the desktop size. The background and actions would all stay the same, only thier presentation would change. If you plugged into a bigger monitor, new layout specific for that screen size.

Afterwards

We stayed for the evening social event, I met up with a couple of people and then began the long drive home.

This is the end of part 1. Part 2 coming up soon ....

Where did all the creativity go?

I'm watching TV this evening and a commercial comes across the screen. A village is being invaded, a boy with an overly large sword, long hair, and no shirt. "Oh no" I exclaim. My wife doesn't know yet. More scenes, poor english, more shirtlessness, more massive sword, then the title. My wife now exclaims "What!". It's "Conan the Barbarian."

What happened to creativity? Have all the creative writers died and my generation is just too lazy to think up anything original? Are executives forcing cheap remakes to keep up the bottom line?

Part of me thinks I'm just being cantankerous before my time, especially when I go looking for information about this topic and I find a similar article in the Christian Science Monitor. All I have to do is think of Top Gun 2 and I realize I'm not being as irrational as I think I might be. Do we need a "Top Gun 2"? Couldn't you have made a fighter plane movie in this era without relating it to Maverick? Oh I forget, "Stealth" was such a epic failure that obviously it wasn't the poor plot idea or so-so acting, it was the fact Maverick wasn't in it.

T-minus 6.5 hours

48 hour film project starts in six and a half hours .....

University of Florida's VIVO running 1.2 SDB

We did it, after a couple of late nights and alot of planning, the University of Florida's VIVO is running 1.2 in SDB. SDB, or Sparql Database for JENA, has been best described to me as Jena's second iteration on semantic data stores. The first iteration, RDB, stored two tables for every model, one for general statements and one for reified statements. The SDB focuses on a single table for all triples. Main model triples are stored in "Triple" while named graph triples are stored in "Quad". We're very excited to see the new performance with VIVO 1.2. You can see the site here, vivo.ufl.edu.

Harvester Webinar: 1st Presentation

Today was the first day of the Harvester Webinar's. We've got about 34 people scheduled, and they evenly split between Tuesday 11am and Wednesday 1pm EST. The webinars are about 2 hours long, focusing on the basics of setting up the Harvester Library and using building a script for harvesting people and publications into VIVO.

Tuesday's webinar was well attended, 7 non-harvester team connections were watching the webinar with several of the connections have 2+ people. Alot of great questions and good feedback. Primary concerns seem to be performance and perhaps usability/accessibility. We'll be addressing these over the next 3 iterations as we begin to integrate with the VIVO code base for User Interfaces. We'll be posting everyone's questions and posting a copy of the webinar on vivo.sf.net very soon.

Fitness like a geek

I've read all kinds of storied about geeks finding new ways to fitness. My favorites are the stories about people building fitness devices that power their game playing. Treadmills that move WoW characters, bikes that power game systems. Its all about being healthier while staying true to your self.

Tonight I embark on exercising with my Xbox using "My Shape, fitness evolved". I loved the Wii fit, but pulling that balance pad out all of the time has gotten to be annoying. Now that I own a Kinect, its time to try out the camera for fitness. Desiree has already used it and its supposedly a very good workout. Here goes nothing.

VIVO in the Open

The VIVO project has always been "open source". However, as the project progresses we are moving from just "open source" to "open source open development". This is not a simple tasks, with openness brings concerns of accessibility. Many projects are open source and open development, but some lack the tools to enable passerby's to become engaged in their project.

What do we need to do to be open and accessible? Open information? A team at the UF Libraries is migrating data from Cornell's confluence and UF's redmine to the sourceforge wiki. The JIRA ticketing system used by the VIVO teams has been opened for anonymous viewing. Open Communication? VIVO has its an IRC channel on freendode, #VIVO. We also have open listsservs on sourceforge that are archived so anyone can read and search.

All of these open avenues help us as a project be more accessible, but that doesn't mean the information is easily accessible. It will be interesting in the next few weeks, months, even years as the project grows. What is required to make our project more accessible? Is it better organization, better tools? Only time will tell.

VIVO Harvester 1.0

The VIVO Harvester went live as 1.0 yesterday! Its been a long process going from a simple concept of ETL into a library of tools for moving data from various formats into a RDF data store. We still have a long way to go and many tasks on the table: integration into the VIVO GUI, inclusion of more National data sources, better disambiguation algorithms. Check out the release announcement.

The VIVO Harvester team is pleased to announce the availability of release 1.0 of the VIVO Harvester. The release is the result of alot of hard work on the behalf of our wonderful testers and contributors. Many thanks is given to all of your for help. This release brings a major rework and upgrade from the 0.7 series of the harvester. See the included changelog inside the package for full details. There is a growing set of documentation to be found on the sourceforge wiki. A great place to start learning more about the harvester can be found here:

http://sourceforge.net/apps/mediawiki/vivo/index.php?title=VIVO_Harvester
http://sourceforge.net/apps/mediawiki/vivo/index.php?title=Harvester_User_Guide

You can find the tarball and debian package on sourceforge here; http://sourceforge.net/projects/vivo/files/VIVO%20Harvester/harvester-1.0.deb/download
http://sourceforge.net/projects/vivo/files/VIVO%20Harvester/harvester-1.0.tar.gz/download

As always, the team is available for questions and feedback via this list or IRC -- http://webchat.freenode.net/?channels=VIVO&uio=Mj10cnVlJjQ9dHJ1ZSY5PXRydWUmMTE9MjM2JjEyPXRydWUe5y

Finally, if your curious about testing VIVO or the harvester without the hassle of setting up a new server, the updated version of the Virtual Appliance is now available. It contains both VIVO 1.2 and the VIVO Harvester 1.0.
http://sourceforge.net/projects/vivo/files/VIVO%20Virtual%20Appliances/VIVO-v1.2-VirtualBox-Standard.tar.gz/download

Fitness and Stress Loss (Part 1)

Since my shoulder surgery and physical therapy I've begun to go to the gym in the mornings. No one should see the side of 6am I've been seeing, but if it reduces my weight, increases blood flow, reduces stress and overall makes work more bearable; I'll do it.

I've been out for a couple weeks due to the Cornell/Indiana visit and the cold I've been percolating. Monday I start back again and I thought it time to begin to document my progress. Documentation can be reinforcing and enlightening as I try to reach my goals. So today I take my weight, and throughout the weeks, each day I'll blog about my attempts at increasing my activity while fixing my diet (not dieting, two seperate concepts). Every so often I'll check the scale and see if there has been progress.

Today's starting weight: *cough*272*cough*