Wednesday, July 17, 2024

Why innovation is rare in big companies

I have worked for small and large tech companies, and as I was sitting through the latest training course this week on the importance of obtaining patents for business purposes (a distasteful but necessary thing in the modern litigious world), I was reflecting on how much more innovative smaller companies tend to be. This is, of course, the general perception also, but it's really reflective of the realities of how companies operate which motivates this outcome, and this is fairly easy to see when you've worked in both environments. So, I'm going to write about this a bit.

As a bit of background, I have two patents to my name, both from when I worked in a small company. Both are marginally interesting (more so than the standard big company patents, anyway), and both are based on work I did in the course of my job. I didn't strive to get either; the company just obtained them after the fact for business value. I have no idea if either has ever been tested or leveraged, aside from asset value on a spreadsheet.

Let's reflect a bit on what it takes to do projects/code at a typical big company. First, you need the project to be on the roadmap, which usually requires some specification, some amount of meetings, convincing one or more managers that the project has tangible business value, constraining the scope to only provable value, and getting it into the process system. Then you have the actual process, which may consist of ticket tracking, documents, narrowing down scope to reduce delivery risk, getting various approvals, doing dev work (which may be delegated depending on who has time), getting a PR done, getting various PR approvals (which usually strip out anything which isn't essential to the narrow approved scope), and getting code merged. Then there is usually some amount of post-coding work, like customer docs, managing merging into releases, support work, etc. That's the typical process in large companies, and is very reflective of my current work environment, as an example.

In this environment, it's very uncommon for developers to "paint outside the lines", so to speak. To survive, mentally and practically, you need to adopt to the process of being a cog who is given menial work to accomplish within a very heavy weight process-oriented system, and is strongly discouraged from trying to rock the boat. Works gets handed down, usually defined by PM's and approved by managers in planning meetings, and anything you do outside of that scope is basically treated as waste by the organization, to be trimmed by all the various processes and barriers along the way. This is the way big companies operate, and given enough resources and inherent inefficiencies, it works reasonably well to maintain and gradually evolve products in entirely safe ways.

It is not surprising that this environment produces little to no real innovation. How could it? It's actively discouraged by every process step and impediment which is there by design.

Now let's consider a small company, where there are typically limited resources, and a strong driving force to build something which has differentiated value. In this environment, developers (and particularly more senior developers) are trusted to build whatever they think has the most value, often by necessity. Some people will struggle a lot in this environment (particularly those people who are very uncomfortable being self-directed); others will waste some efforts, but good and proactive developers will produce a lot of stuff, in a plethora of directions. They will also explore, optimize, experiment with different approaches which may not pan out, etc., all virtually unconstrained by process and overhead. In this environment, it's not uncommon to have 10x productivity from each person, and only half of that work actually end up being used in production (compared to the carefully controlled work product in larger companies).

But, small companies get some side-benefits from that environment also, in addition to the overall increases in per-person effective productivity. Because developers are experimenting and less constrained by processes and other people's priorities, they will often build things which would never have been conceived of within meetings among managers and PM's. Often these are "hidden" things (code optimizations, refactoring to reduce maintenance costs, process optimizations for developer workflows, "fun" feature adds, etc.), but sometimes they are "interesting" things, of the sort which could be construed as innovations. It is these developments which will typically give rise to actual advances in product areas, and ultimately lead to business value through patents which have meaning in the markets.

Now, I'd be remiss to not mention that a number of companies are aware of this fact, and have done things to try to mitigate these effects. Google's famous "20% time", for example, was almost certainly an attempt to address this head-on, by creating an internal environment where innovation was still possible even as the company grew (note: they eventually got too large to sustain this in the face of profit desires from the market). Some companies use hackathons for this, some have specific groups or positions which are explicitly given this trust and freedom, etc. But by and large, they are all just trying to replicate what tends to happen organically at smaller companies, which do not have the pressure or resources build all the systems and overhead to get in the way of their own would-be success.

Anyway, hopefully that's somewhat insightful as to why most real innovation happens in smaller companies, at least in the software industry.


No comments: