Jun18

Some Fun Web Part Errors from Today

So, many issues I had to resolve today were typical over-rushed developer stuff: non checking for nulls, not testing with users that don’t have MySites, yadda yadda.

 

Two errors stood out as things I'd like to help people avoid doing.

 

First, the developer of the web parts I am fixing failed to make a distinction between System.Web.UI.WebParts.WebPart and Microsoft.SharePoint.WebPartPages.WebPart. They are almost identical, but each implements Web Part Properties in code slightly differently.

using com = System.ComponentModel;
using aspnetwebparts = System.Web.UI.WebControls.WebParts;
using wsswebparts = Microsoft.SharePoint.WebPartPages;

...

// for System.Web.UI web parts
[aspnetwebparts.Personalizable(aspnetwebparts.PersonalizationScope.Shared)]
[aspnetwebparts.WebBrowsable(false)]
[com.Category("Configuration")]
[aspnetwebparts.WebDisplayName("Parent Category ID")]
[aspnetwebparts.WebDescription("Determines the Parent Category from the Categories List on the Taxonomy Administration site")]

// for WSS web part
[wsswebparts.WebPartStorage(wsswebparts.Storage.Shared)]
[com.Browsable(false)]
[wsswebparts.SPWebCategoryName("Configuration")]
[com.DisplayName("Parent Category ID")]
[com.Description("Determines the Parent Category from the Categories List on the Taxonomy Administration site")]

This was probably a copy-pasta error, but he used the System implementation instead of the SharePoint one. As a result, none of these web part properties actually showed up where users could change them.

 

To be honest, I used to make this mistake all the time, especially when copying properties from other web parts I had written. These days, I just suck it up and define all of the attributes so I don't have to deal with this issue.

 

That was actually good news, because it means that maybe with a little more work, this solution will be about 10x as configurable as it is now.

 

The second thing almost drove me crazy, because it was very similar to a problem I saw a year ago (or more) and I couldn’t remember what it was. It turned out in the end this was a slightly different issue, but I digress. The problem was that when you have a Tool Pane Editor (basically a fancy custom Web Part Property) and it is used to set a WPP that’s also visible in the browser, the WPP will overwrite the changes that the TPE makes. So for example, you pick "IT Apps" in a dropdown TPE. The TPE goes and looks up the List ID for "IT Apps", and comes back with “15” and saves that to the WPP called Parent ID, but since Parent ID is also visible in the web page and has a value of “” (empty) the empty value then overwrites the “15” and you’re back where you started. The solution here is to hide the WPP by setting Browsable/WeBrowsable = false, thus proving it’s not how much code it takes to fix a bug but finding the correct line to change. J

 

To the developer's credit, I was the one who set this to true, because I was trying to make it visible so I could troubleshoot the ToolPane picker. Oops!

 

(By the way Ted, I hope you're settling in nicely at DOE. <grin>)

Published: Jun-18-09 | 0  Comment | 0  Link to this post

Mar30

Dynamic Data Source Controls for Your Data Form Web Parts

In this very, very, *very* old (but still interesting) ASP.net article on MSDN Magazine, I found the following clever code snippit that uses a temporary file to create a user control dynamically from a string:
 
TemplateColumn bc = new TemplateColumn();
String tmp = Session.SessionID + ".ascx";
StreamWriter sw;
sw = new StreamWriter(Server.MapPath(tmp));
sw.Write(strLayoutCode);
sw.Close();
bc.ItemTemplate = Page.LoadTemplate(tmp);
grid.Columns.Add(bc);
File.Delete(Server.MapPath(tmp));
 
This is indeed pretty neat, but it turns out it's not necessary. You can do the same with the following:
 
Page.TemplateControl.ParseControl(strLayoutCode);
 
I discovered this interesting fact after realizing that SharePoint's DataFormWebPart is doing some intriguingly useful (but obfuscated) stuff with DataSourceControls and the SQL database. Basically, in the SharePoint Designer, whatever controls you put inside the DataSources tag - using the design pane, I would assume - get serialized into a BLOB in the WebParts table of your WSS Content database. At run time, they get loaded from the database on the first call to get the DataSources collection, which typically happens during CreateChildControls.
 
I tried forcing a load by explicitly setting DataSourcesString within code, but it blew up because the parser did not understand the tag prefixes in the code that I inserted. It was then that I realized that SharePoint is leveraging ParseControl(string), and after digging in the database, I was able to see that SPD must be programmatically determining what namespaces to register when it saves the data to SQL by way of the DataSourcesString property.
 
This is useful to me because for a long time I have been thinking that SharePoint, and in a greater sense ASP.net needed a way to allow data source controls to be defined more flexibly and in a centralized location. Sure, you can do it programmatically. But, the control parser gives you a fairly efficient way of doing it using a repeatable and flexible process, and you won't be limited to keeping the controls on the filesystem.
 
If you want to leverage this functionality without giving up the other benefits of the Data Form, check out SPARK project over at CodePlex. It includes Flex Data Form, a version of the DFWP that inherits all the same features as the original, but allows you to set the DataSources string and some other properties from within the Web UI. (It also has some other nifty features as well.)
 
Update: Looks like I am not the only one thinking along these lines. Too bad his comments are closed, because his ideas are good ones and he deserves the Whuffie. :-)
 
More Data Form goodness, especially a decent explaination of the OOTB method for overcoming the list GUID issues that plague migration of the DFWP.
Published: Mar-30-09 | 0  Comment | 0  Link to this post

Mar01

Setting up TFS Server and using VSSConverter with VSSConverterGUI

 
I spent the weekend installing and configuring TFS Server 2008. I learned a lot, but mostly it was just a pain. In the end we did get things working properly.
 
After install, I wanted to port over our existing VSS from a past development project. I found this article, Migrate VSS Projects to TFS Projects, which was indeed a useful walkthrough, although it missed some important steps that were necessary for me.
 
First, the VSSConverterGUI is compiled only for Visual Studio 2005. Luckily I did have this older version lingering around still. I installed the TFS Explorer for VS2005 using the link I found on the Team System Blog. Note that the image file is in a wierd format, so I had to install Magic ISO in order to extract it; I really didn't feel like wasting a blank CD for this.
 
You put the GUI exe in the same folder as VSSConverter.exe and it will actually work, which is great. From there, I had to just browse to my VSS and TFS server, which worked very well.
 
The next issue is that I could not get teh User Mappings part of the form to work at all. So, instead, I saved the setting.xml and closed the application, and ended up rolling my own UserMap.xml in the same folder as Settings.xml using the example from the blog walkthrough above. I was able to use SourceSafe 2005 Admin to view the users so the mappings were pretty much intuitive. The other thing to note is the the tool likes it if you save both these XML files to the VSS folder, not the Common7 IDE.
 
Finally, I got the message that says "TF60032: The VSS Converter requires Visual SourceSafe 2005 or later to run. Please install Visual SourceSafe 2005 or later and try again. " like this poor fellow and this one. This was very weird because I *did* have SourceSafe 2005 installed on the very same box and was using it. So, I did some digging and thanks to this article, was able to get it working by adding the following keys to my registry using a REG import file.
Windows Registry Editor Version 5.00


[HKEY_CLASSES_ROOT\SourceSafe]
@="Microsoft Visual SourceSafe Automation"
[HKEY_CLASSES_ROOT\SourceSafe\CLSID]
@="{783CD4E4-9D54-11CF-B8EE-00608CC9A71F}"
[HKEY_CLASSES_ROOT\SourceSafe\CurVer]
@="SourceSafe.8.0"
Finally, the import ran correctly and now all our source code in in TFS!
 
Published: Mar-01-09 | 0  Comment | 0  Link to this post

Feb23

SharePoint Code Published on CodePlex

After thinking this over for a while, and hiring a new employee, I've decided to truly make my SharePoint library both a more accessible and more collaborative effort by publishing it using CodePlex.
 
 
The SPARK project includes many of the examples published previously in articles from this blog, so those who have commented that you received 401/403 errors here should go there and download the source code. Just search through the files in the Behemoth.SharePoint project it to find the classes mentioned in the blog.
 
Why Behemoth? Well, because at least right now, the code is a big unstoppable monster. There's tons of code that needs to be ported over, (re)tested, and documented. Like any mythic beast, it is powerful and also a little scary. Plus the name fits with the theme of my company Colossus Consulting.
 
I'll be doing more additions and documentation in the coming weeks, so I hope if some of you find the code informative or useful that you'll leave me some love in the comments and reviews. (I also accept beer from those in the DC area.)
Published: Feb-23-09 | 0  Comment | 0  Link to this post

Dec04

My Four Day Trip through Wonderland: Fixing Windows Live Authentication in the CKS EBE

At first, I was thinking about naming this blog post "Can It Really Be That Stupid?" After a few seconds, I realized the answer is "Yes.", got over it, and moved on. For one thing, I would have to decide which thing was *that* stupid, and that's just asking too much.

So, there are a number of people out there using SharePoint with WLA. It seems like a cool concept. Just use the WLA membership and role providers the way you would any provider for forms based authentication, and "Voilla!" your friends or business partners on Live Messenger can log in to your site using their Live ID , whereupon you can give them special access.

Last year, the Community Kit for SharePoint released a WLA membership provider designed to be used with the Enhanced Blog Edition. If you're curious - and very brave - you can get the source code from CodePlex here. The solution comes with a readme, which leaves out a couple of minor steps, but Rolf Eleveld has a nice walkthrough with screenshots on his blog.

Oh, if only it had actually been that easy!

Access Denied Errors on Login for LiveAuth-Handler.aspx

After following all the instructions to set up WLA, I tried to log in on the public facing site by clicking the Sign In link, but I kept receiving the "401 UNAUTHORIZED" error. A quick check of the IIS logs reveleaded this as "HTTP 401.5: Denied by custom ISAPI/CGI Web application", which means that the access issue is occurring within SharePoint itself.

There are so many causes for this kind problem, but I remembered solving such an issue about a year and a half ago. Here are some of the things you can try if you are having 401 problems.

  1. Disable Loopback Checking
    This security "feature" is enabled by default with the latest service packs, but it was not really meant to be used in SOA environments like SharePoint. Basically, whenever SharePoint attempts to hit its own web services on the local machine, IIS will block the attempt and cause SharePoint to throw an Access Denied error. The issue described in the Microsoft KB Article is not exactly what I was seeing, but it contains instructions on how to perform the registry hack. In the past I have seen this problem cause unusual symptoms within SharePoint that you normally wouldn't see in a vanilla ASP.net application, so I wouldn't rule it out.
  2. General Tips for 401 Errors in IIS
    You can also try some of the things suggested by this Microsoft KB Article. It has useful links to some diagnostic tools including the AACD 1.0 and FileMon. At the very least, following this approach should help to uncover any typical problems that would interfere with any SharePoint install, with or without CKS.
  3. SharePoint File Permissions
    Make sure you have all the correct file rights set up for the anonymous (IUSR_MACHINE) and application pool accounts. You can use FileMon to test for this (see above). This will include rights to the ASPX files in the LAYOUTS folder, including special anonymous access permissions that are needed for Login.aspx, Authenticate.aspx, and in our case LiveAuth-Handler.aspx. Generally, if you have IUSR_MACHINE in the WSS_WPG group, and the application pool account in the WSS_ADMIN_WPG group, then that ought to do it.
  4. Is Your Provider Working?
    Make sure your authentication provider is set up correctly. In this case, I would double check the setup as per the instructions described with the WLA provider above.

    One way to test this is to add the same provider settings to your internal (Windows Authentication) and Central Administration sites. With these added, you should be able to go into your windows authentication based site and add Live Authenticated Users as a group with rights within SharePoint – even if you can't successfully get the Live ID login process to work.


    Figure: Testing correct setup of WLA Users and Groups from the intranet (Windows Auth) site

    In my case, I added the membership, roleManager, and appSettings nodes to the web.config file located at C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\CONFIG, because I wanted these providers to be accessible to every web application on my development machine. However, I will probably go "by the book" in production.

    Also, note that I have commented out the identity node (namely, impersonation="true") in the public facing (FBA) web.config file, as this is normal practice for FBA sites, but wasn't mentioned in the WLA documentation.

Unfortunately, none of these things I tried actually solved my problem. Every page I would try to visit would redirect me to Authenticate.aspx, and then that would subsequently 302 redirect me to LiveAuth-Handler.aspx?action=login, which would give me the same 401 error.

As a quick aside, let's look at how the login process actually works, because I was thinking about it a lot at the time. We start on the main page of the blog, and if you have Anonymous Access configured in SharePoint and also correctly configured in IIS, then you'll see the default.aspx page with its Sign In link displayed correctly.

As you can see, this link takes you to Authenticate.aspx, which is SharePoint's default page for requiring a login, and the one that is responsible for redirecting you to LiveAuth-Handler.aspx as described in web.config, whereupon, we get the dreaded 401.5 error. Authenticate.aspx seemed to be working as designed, but I was at a loss to explain the error.

My Eureka Moment: Running Out Into the Street without My 401

After a full day of beating my head against the machine I was getting a bit crazy, so I decided to take a break. I put the computer away and went to an Orioles game with a pal of mine. Perhaps my best tip of the day, do not underestimate the value of taking a break. Your brain will keep working on the problem in the background while you recharge your batteries. In fact, I had a great time. Not only was this the first time in over a decade that they won while I was at the stadium (what awful luck!), but they delivered a spectacular grand slam followed by a three run homer, literally crushing the Cleveland Indians.

So, the following morning I was thinking about what is different in the WLA that I have never seen in any FBA based solution before. If I could find the variables, and eliminate them, I might solve the problem or at least discover the cause. One thing that struck me as just-too-stupid-to-think-about was the URL that specified in the loginUrl attribute of the forms tag in the authentication section of web.config.

"What if," I thought, "however unlikely it may be, there is something in the .NET authentication provider that refuses to recognize the login page simply because it can't handle the query string at the end?" Well, it sure seemed like the kind of shortcoming that would typically sneak into code, but I had never heard of such an issue. Still, there are lots of patches both for .NET framework and SharePoint, so maybe I'm on a patched version that has this rare problem.

So, I tried a little experiment. I took out the query string, like so.

Click the Sign In link again, and…

"Wow! There's no way it can be that simple." I put the query string back into the web.config, then tried to log in again, just to be sure.

In fact, if you take the query string out of web.config, but you add it manually on the address line of your browser, it will work as expected. The result redirects you to the Windows Live web site without any 401. Note that I've made some improvements to my source code. If you manually type the login URL on the address bar using the code as downloaded from CodePlex, you are much more likely to see some kind of NullReferenceException instead of a successful 302. But my point is that it will actually run your code.

The Final Verdict: Ye Not Guilty

So, then, this answers how to fix the problem, but doesn't identify the cause. Is this an issue with the .NET framework or maybe SharePoint itself? Well, that might be possible, but I thought it was unlikely given that these products have probably gotten some pretty thorough testing, especially compared to CKS and the WLA provider. But, I was on a mission. I wanted to make this work, and I had a clear path to explore, so there was no time to ponder this in depth.

What should've also crossed my mind at the time - but didn't - is that this issue might have been caused by code running in CKS itself. I ran into some other issues in the following days, but after everything blew over, I did some experiments to try and prove that CKS EBE itself was not at the heart of this particular problem. (Turns out I proved myself wrong.) To test this, I tried changing the settings for Secure System Pages in the CKS EBE Settings (a.k.a. Blog Settings).

There were two takeaways from this that finally explain this issue in black and white.

Lesson #1
The first is that just because you change your Blog Settings on the internal (Windows Authentication) site, does not mean they will automatically change on the public facing (FBA) site. While trying to make changes to the setting above, I learned that you need to make changes to Blog Settings manually to both sites. This is actually pretty cool, because it means you can select different themes for each and change some settings as appropriate. Also, it makes sense, since these blog settings are stored as a section in web.config for each site. But it is also a pain, especially if you have already configured FBA on the public site and you can't actually login to make settings changes.

Lesson #2
Second, I learned that everything I did from here on out is – at least partly – unnecessary. To work around this issue you simply need to turn Secure System Pages off. This will eliminate the problem with the login page having a query string. Of course, I happen to like this setting and want to use it on my public facing site, so I'll be working with this code to make it work properly with WLA.

Ever Deeper Down the Rabbit Hole

Now that I've revealed "whodunit" in this little mystery, let me jump back. Instead of finding the exact cause right away, I would spend the next two days trying to make the login page work correctly without the query string. I went through a lot of code changes to liveatuh-handler.cs in the attempt, and I learned a lot. Because this is not the end of the story, I plan to talk about these things just a bit before I'm done, but nobody will blame you if you skip all that and just get the code. [TODO: publish and link to code please]

So my first approach was to try and rewrite liveauth-handler.aspx so that it could just tell that you were going from Authenticate.aspx and redirect you to the Windows Live site automatically without being told via the query string. To some extent, this worked, but it took some messing around the find the right commands, and the HTTP_REFERER was not always what I expected it to be.

Here's a screen shot from my early code:

You can see at this point that I had already done some work to eliminate "Object Reference Not Set" exceptions. That's pretty typical of code written in a rush. You get it working to the extent that you need it to. Since you have other stuff to do, there isn't time to imagine every way in which it might go outside that corral, you move on. Unfortunately, it results in code that breaks easily, which it did for me many times.

You can see in these shots of the debugger as I was digging around for the HTTP_REFERER variable within the page Request object.

Its value would typically be something ending in "default.aspx" or "authenticate.aspx" whenever I was clicking through from the Sign In link. It would be easy enough to tell that the referrer was on the same server using a string comparison or RegEx, but fortunately the .NET framework has provided a couple of handy properties in the HttpRequest class that help to isolate the host name. These are Url and UrlReferer as seen below.

Interestingly, page redirects from the WLA site would have no referrer at all. I would've preferred something like "https://login.live.com/ppsecure/post.srf?appid=000000004400143A&alg=wsignin1.0..." Alas, no, they are just blank. This makes them indistinguishable from URLs entered on the address bar, and that made testing a nuisance.

Despite this, I was able eventually to muddle through. There were a number of setbacks, like this one:

This issue, or some other error, will happen when you have Secure System Pages turned on. (The exception message is my code and not part of CKS EBE or WLA, because I thought it would be helpful for other developers.) I did some experimentation to prove that when Secure System Pages is turned off, the new users will be automatically redirected to liveinfo.aspx as they should be. Also, the issue does not affect users who have already entered an email address in their Blog User Profile, because those users are typically redirected to a generic (unsecured) page such as default.aspx instead.

What's going on here is that Windows Live returns with a valid session ticket as an HTTP form post, but since the user does not yet exist in SharePoint or have any rights other than those granted to Live Authenticated Users, they get redirected from liveinfo.aspx to the liveauth-handler.aspx login page, probably by way of authenticate.aspx. (In the original source code from CodePlex, this would've forced their session to be logged out.) The redirect causes the contents of Request.Form to be lost, which is the means by which WindowsLiveLogin.ProcessLogin() method receives data from the Windows Live website. Even though you'd think that such an internal referral should have some kind of value in Request.UrlReferrer, its value is in fact null. The code, as I had written it at this point, failed because this condition is a contradiction. On the one hand it thinks it is supposed to process some kind of result from Windows Live, but that information has been dropped.

In a nutshell this means that any user who hasn't provided an email address is going to get some kind of error after they log in, and it is very likely that such an error will cause the authentication process to break down before all the cookies have been properly set. Looking at all this strange behavior, I have decided that the entire flow for the page needs to be streamlined and bulletproofed. (Though as of December I still haven't gotten around to it.) It's too convoluted and unreliable when conditions fall outside the norm. In the long run, the issue can be corrected entirely by fixing the code for BlogHttpModule.cs in the CKS.EBE solution.

But, let's talk a minute about the process by which I came to discover that this is what was going on, because it did not come easily.

To have results come back from the Windows Live site, and not have any form attached to them, was very confusing to me. I wanted to try and prove that the form was in fact coming back across the wire, and then being lost someplace else. To this end, I downloaded a very useful tool called EffeTech HTTP Sniffer which I have used on many passed projects to gain insight into what SharePoint is doing behind the scenes. It is my opinion that an HTTP Sniffer of some kind should be included on every SharePoint developer's installation checklist.

I was unable to get any meaningful data from the HTTP Sniffer though, which returned only a generic HTTP request and a GIF for the passport logo. Upon post-back from the Windlows Live server there was not showing up in the trace. So, I tried their EtherDetect product to see if I could figure it out. Here is an example of what came back.

Digging into these, I was able to determine that my server was opening a random port that connected to their server on port 443. Several tests concluded the port was random, and so I was unable to add it to the list of ports in the HTTP sniffer, and everything I saw in the Ethernet sniffer was of course encrypted. Unfortunately, this is a failing of the EffeTech HTTP Sniffer. The MSN Sniffer was totally useless for WLA. But, at least I was able to determine that there was some kind of data coming back.

Eventually, I was able to blunder into the solution of manually adding an email address to my entry in ProfileList, and that got me around the problem with redirection to viewinfo.aspx. This is not a good final solution, but I was able to move on to the next problem.

Okay, so now I could see that I had a form and I was able to step through code to the point where I could see that SetAuthCookie was executing. But, as far as I could tell, there was no cookie and no authentication within SharePoint going on at this point.

By this time, I may have been chasing foo-fighters. I had been a while in the weeds, and the default theme for CKS EBE provides no visual indication that you are signed into SharePoint as any particular user - something I have since changed in my own blog site as you can see at the top of this page. For this reason, I recommend that you create your own theme that makes use of the wssuc:Welcome control to tell you who is logged in. It could very well be that for some time I was actually successful at authenticating and did not know it, because Live Authenticated Users was not assigned any special rights. Note that even if you add this group to the Contributors, it will not automatically be able to create Blog Posts.

Another thing I tried was to remove the domain and path properties from my forms tag in web.config, but I have since put them back so I know they are not to blame in this case. I also tried changing the name of the form from "livelogin" to "LiveID", but I have also changed this back without any effect. For the time being if there were authentication issues around cookies, I have been unable to reproduce the problem.

Ultimate Solution: Fix Secure System Pages

Although I did eventually end up reverting to the use of a query string for the liveauth-handler.aspx page, I wanted to clean up the code somewhat anyway. I also implemented a fix for the BlogHttpHandler.cs that resolves many of the issues described above, and could probably be improved upon even more. Given time I would like to make the list of allowed system pages configurable from an XML file in the Feature (or maybe even web.config itself).

You can download the code here. I only provide the files in question, and I'll submit these issues to CodePlex as soon as I get the chance, but I wanted to make them available here for those experiencing similar problems. Enjoy! [TODO: publish code and provide link]

Published: Dec-04-08 | 0  Comment | 0  Link to this post

Dec01

What Tool Should You Use to Deploy SharePoint Projects?

At this point, I have tried them all. I hand rolled my solutions; used early versions of WSPBuilder; struggled with the interface for VSeWSS 1.0, 1.1 CTP, and 1.1 final; and even tried a few offbeat solutions.
 
Until this summer, every method had serious problems that made packaging and deployment problematic. Revisiting my posts, I see that I never mentioned this before, but I owe props to my colleague Ted Calhoon who turned my on to the WSPBuilder Extensions. I switched over, and I never looked back.
 
 
They are a bit counterintuitive, and the commands to flush out the folder structure are well buried, but once you get the hang of them, they are the best around - at least until someone comes up with something better.
 
In future, I'll do a walkthrough.
Published: Dec-01-08 | 0  Comment | 0  Link to this post

Jan30

STSDev 1.1: Another Visual Studio SharePoint Development Tool

Well now in addition to WSPBuilder and the VSeWSS, you now have yet another method you can use to build out your SharePoint projects.
 
 
I have not yet downloaded and tested this tool, but they seem to have some pretty good tutorials for it and have made the source available, which is a HUGE plus over VSeWSS.
 
I'm going to give using it a try some time this week and maybe do a brief review of it. But for now why not give it a try yourself?
Published: Jan-30-08 | 1  Comment | 0  Link to this post

Jan25

I Thought I Was Done, But I Had Another Thought

So, I guess I am not finished my brain dump. Given my previous post, what do we know, and what's to be done about it?
  • Because people don't accurately remember times for compelting small tasks, programmers should measure their own productivity.
  • This is boring work and should be done using software.
  • Programmers will resist having their true productivity measured by management, so they should have the tools to do it for themselves.
  • The measurements of one programmer are utterly useless for anyone other than that programmer, as code production can differ by a factor of 10:1 depending on the individual and circumstances.
  • If programmers do not share their individual data, then it isn't possible for management to make accurate estimates.
  • This information is most valuable to the programmer themselves, because it would improve estimaties.
  • Management is more concerned about creating reliable estimates than in comparing individual output.
  • In programming, low level tasks of different types are not necessarily interchangeable in terms of time. Individuals have different specialization even within a lagnuage or a framework.
What's this mean? Should we just give up? That probably won't fly, but here are a couple ideas that might help.
 
It would probably be better if we don't ask programmers themselves to provide estimates. Instead ask a programmer to describe the tasks involved in getting a project done and then use reliable large-scale data to determine how long a project should take given a average level of productivity.
 
Forrester or the like may have done some of this research or found studies done at universities. I have read some general studies on this topic that were produced by Corporate Executive Board during my time on their SharePoint project in 2003, but I don't know if they have drilled down on specific activities. Another good source would be to follow Fredrick Brook's references and see where they lead and if there have been any modern updates.
 
Why would programmers not like this done within an individual company? Well, it's basically along the same lines as truck drivers having GPS in their trucks or weight detectors under the passenger's seat. I think that while the programmer who knows or believes that he is 10x more productive than his peers would welcome such a system, there would a majority of others who wouldn't want to give that person more ammunition or ego fuel. In the end, I think even the super-programmer knows that it is better to have solidarity with his brothers if he expects their help later on. 
 
Sidebar: Some of the lady programmers out there may have noticed my assumption that my hypothtical programmer is male. Well you ladies are unfortunately still very rare in our feild, though I do wish that would change. In any case, being so rare, you're probably going to get our help no matter what you say or do, so it would be survivable I think to lord your superior productivity over the group and demand that everyone count their code lines. (Of course, I hope by now you all understand that I am kidding, right?)
 
We could probably all benefit from a large scale (open source?) project that would allow the entire IT community to provide details about how long tasks take. There could be plugins for popular tools like TFS/vs.net for task management to make participation easier.
 
If the data is collected in large anonymous sets and each developer has access to their own numbers, then it would remove much of the resistance to the idea of managment grading everyone. This would be much more like the way insurance groups are created. You're boss doesn't have access to the information about your health that the insurance company uses to determine how much they will charge him or her for your medical plans. (Actually, the insurance company doesn't get the whole picture either, but that's not the point right now.)
 
To make such an idea work it would have to be easy, it would have to be reliable (fake proof), and it would have to be fun.
 
Food for thought. I'll come back to this again some other time.
Published: Jan-25-08 | 0  Comment | 0  Link to this post

Jan25

Why Are Programmers' Estimates Usually Wrong?

Okay, I don't have a ton of time for this post today, but I wanted to get it out of my headspace so I can focus on other things. I'll just be summarizing my ideas here very briefly, so if you can't keep up I'll understand. I plan to come back and write something more formal later on.
 
Here's the basic idea. I've been thinking about why programmers tend to come up with estimates that are always too short. This is a problem that is pretty well known in the IT community.
 
Many programmers apply the Scotty Rule to their estimates (take what you think it will take and multiply by three). I can tell you that I've tried that, and it's still very possible to come up with an estimate that is still too short.
 
Fredrick Brooks has gone into detail on this topic in several of his essays. If you aren't familliar with him, I suggest you run out and buy a copy of the Mythical Mon-Month (Wiki, Amazon) right away. There are also some discussions of the problem in the book Dreaming in Code (Amazon), which I have only half-read.
 
Here's a summary of the discussion I've had with colleagues about this, based partly on the sources above, and partly on my own and others' experiences:
  • Programmers aren't capable of coming up with solid estimates because they are too optimistic. Their nature leads them to believe the impossible is possible and to chronically underestimate the level of effort required for any task.
  • However, without this (psychotic? disassociative?) optimism, they would not be capable of taking on the burden of creating large and extremely complex systems that must function [nearly] perfectly [mostly] all of the time.
  • Further, estimates are generated at the start of a project, when very little is actually known about what specific small scale tasks will be required. Even if the design is known, the detail level where the actual work lies is generally not fully understood.
  • Update Added as a caveat to the above item: obessive listmaking does not help. It is inevitable that a single person drilling down into the required tasks will overlook something, somewhere, and that thing will likely be important. Peer review or collaboration will only help this to the extent that people are actually willing and capable of reading such a detailed laundry list. Imho, the best lists would be generated by logging actual tasks done on several similar jobs, and combining the results into a superset - with notes about when and why certain tasks were only sometimes necessary.
  • A good programmer would rather try to do something that is hard.
  • There is a real transition along the lifespan of a project, from a learning and creative process at the start to a controlling and list maintaining process near the end. Programmers who are talented at design and early development are not necessarily well suited to do stabilization, but are frequently required to.
  • We actually know a lot about estimating software and managing development projects, but for the most part we don't actually act on that knowledge. Brooks is correct in asserting that his book is the software development Bible (everyone reads it, nobody follows it). In 30+ years since its publication our tools have gotten better, but we really haven't learned to actually *do* things much differently.
  • Yadda, yadda, the list goes on...
I am adding an item to this list today, after having read this New Scientist article about research into how we perceive the passage of time. My premise is that one of the important reasons programmers can't accurately give estimates is that they can't accurately remember how long any particular low level task has taken. To support this idea with facts will require some more research, of which there isn't that much available.
 
But, I think there might be enough to support the idea that we can chronically underestimate how long a future task will take if we chronically mis-remember it as having not taken very long in the past. In other words, being in "the zone" or acheiving a flow state (which are good things, right?) can make it harder for you to remember accurately how long a task actually took you.
 
Perhaps some enterprising student of Management Science will stumble on my blog and figure out ow to arrange the experiment and get some funding.  For now, I'll have to keep digging until I have exhausted the pool of data that is out there today.
 
If this theory is true, then it is possible that estimates could be greatly improved by programmers simply keeping detailed logs of their activity. Sadly, that doesn't seem to be consistent with the "flow state" behavior and might undermine it. I am thinking that software might provide an answer for how to gether the data without disrupting concentration.
 
For example, could we use TFS to measure time taken for work items, and how might one incroporate such a tool into development so that it is a true habit and not disruptive to the creative process? I guess I'll have to answer that some other time though.
Published: Jan-25-08 | 0  Comment | 0  Link to this post

Jan24

To Heck With What I'd Like, How About What I'd *Love*?

Congratulations, Alara!
 
So, congratulations to Alara for landing a pretty sweet gig today. She starts on Wednesday, doing business analysis type work for a healthcare company in Elk Ridge - in addition to whatever else it was she's been doing with the city to earn money through her business.
 
The extra money she'll be bringing in from this new contract will really help a lot. Even if it doesn't go past the 2-3 months that it's commited to, it'll teally put a dent in what has become our too-damn-big pile of debts.
 
A lot of people don't like to talk about money, but for me it is not something I am coy about, and I'm generally very up front about it. The truth is that I make better money now than most people out there - more than I ever expected that I could. I'm not ashamed of that, because I work very, very hard and have cultivated some pretty unique skills. (Acutally, I'm not full of myself, so I know it's a lot of luck plus a little talent.)
 
So, when I say that this past year has been very difficult for us financially, please understand that if it was rough for me, then my heart truly goes out to the 90%+ of Americans who bring home less than I do. I grew up in near poverty; my mom took help from her parents, food stamps, and college aid as she raised me by herself. I know how it feels.
 
Lately, raising four kids, it's hard to figure out where the money goes and why we don't keep more of it. Maybe that is just a part of having kids. Or maybe we should both work a little less hard and instead devote that time into managing our money better. Less time spent in setting up the TVPC and a bit more in balancing the household budget might help. I think a lot of it has to do with the simple fact that the dollar just is not worth what it once was. If I had to guess, I would say it has had about the same effect as a 25-33% pay cut.
 
Oddly enough that's not what I wanted to blog about. I just wanted to thank Alara for her hard work, and for taking the stress (and the daycare bills) off of my shoulders a bit.
 
So, What Kind of Job Would I Love To Have?
So, her getting this gig made me start thinking about my job, about raises I did not get, about what I enjoy about my job, what I'd rather take a pass on, and how I really want to be spending my time.
 
So here goes, my wishlist for a dream job:
  • I want to work with SharePoint most or all the time, because it's really cool!

  • But, I don't want to work for Microsoft.

  • I like working on lots of little short-term projects. Veriety in work and in solving different problems for different types of users is exciting. For that reason, I could probably spend my whole life building nifty web parts and showing people how to use them.
  • But, I hate having to think about my commute [or the people I have to work with] changing whenever the project ends.

  • I truly enjoy using technology to help transform a business. Getting only little wins is really frustrating, so buy-in from the top is very important to me. If that means part of my job is to fight for that support, then so be it. Unlike many people, I find debate invigorating.

  • I like doing work on proof of concept and design. Finding out what can be done is fun. Finding out that you *could*, except that you *can't* because there either a) aren't the skills, b) aren't the resources, or c) isn't the time is no fun at all. So, a place where there is a real investment in technology (as opposed to band aid solutions) is a real plus. (Update: Add "lack of political will" to that above list of frustrations.)

  • I would like to either have a very short commute to downtown Baltimore, or else a reasonable train ride to Washington DC that I can do myself. I am tired of relying on my wife for transportation. I wouldn't mind working in my boxers either, but I need an excuse to get out of the house once in a while, and some things are better done face to face.

  • I'd like a job title where the median base salary is in the neighborhood of $150,000/yr. Software Engineering Director looks nice, though frankly I have never seen anybody hiring for that one. They always use terms like Developer, Analyst, or Architect. Sorry, but the pennies don't spend like they used to, and wages don't seem to be going up to meet inflation. I guess I could accept a lower salary for the right perks, but the money is pretty important.

  • Funnily enough, I like managing technical people, and I am good at it too. That's an aspect of work that I miss when being stood up as a lone gunman in consutling gigs. You rarely if ever have the authority to direct a team. Well, at least sometimes you can act as the trusted advisor; that can be nice.
Too bad that nobody will probably ever ask me what kind of job I want. Wouldn't it be great if we interviewed companies instead of companies interviewing us?
 
For your pleasure, here's my resume. If you feel so inclined, tell me if you think I am qualified for the job I am describing, or for that matter if it even exists.
 
Resume in XPS Format (It's like PDF only Microsoft-ier)
 
Published: Jan-24-08 | 0  Comment | 0  Link to this post

 Next >>
`