Why innovation is rare in big companies

I have worked for small and large tech companies, and as I was sitting through the latest training course this week on the importance of obtaining patents for business purposes (a distasteful but necessary thing in the modern litigious world), I was reflecting on how much more innovative smaller companies tend to be. This is, of course, the general perception also, but it's really reflective of the realities of how companies operate which motivates this outcome, and this is fairly easy to see when you've worked in both environments. So, I'm going to write about this a bit.

As a bit of background, I have two patents to my name, both from when I worked in a small company. Both are marginally interesting (more so than the standard big company patents, anyway), and both are based on work I did in the course of my job. I didn't strive to get either; the company just obtained them after the fact for business value. I have no idea if either has ever been tested or leveraged, aside from asset value on a spreadsheet.

Let's reflect a bit on what it takes to do projects/code at a typical big company. First, you need the project to be on the roadmap, which usually requires some specification, some amount of meetings, convincing one or more managers that the project has tangible business value, constraining the scope to only provable value, and getting it into the process system. Then you have the actual process, which may consist of ticket tracking, documents, narrowing down scope to reduce delivery risk, getting various approvals, doing dev work (which may be delegated depending on who has time), getting a PR done, getting various PR approvals (which usually strip out anything which isn't essential to the narrow approved scope), and getting code merged. Then there is usually some amount of post-coding work, like customer docs, managing merging into releases, support work, etc. That's the typical process in large companies, and is very reflective of my current work environment, as an example.

In this environment, it's very uncommon for developers to "paint outside the lines", so to speak. To survive, mentally and practically, you need to adopt to the process of being a cog who is given menial work to accomplish within a very heavy weight process-oriented system, and is strongly discouraged from trying to rock the boat. Works gets handed down, usually defined by PM's and approved by managers in planning meetings, and anything you do outside of that scope is basically treated as waste by the organization, to be trimmed by all the various processes and barriers along the way. This is the way big companies operate, and given enough resources and inherent inefficiencies, it works reasonably well to maintain and gradually evolve products in entirely safe ways.

It is not surprising that this environment produces little to no real innovation. How could it? It's actively discouraged by every process step and impediment which is there by design.

Now let's consider a small company, where there are typically limited resources, and a strong driving force to build something which has differentiated value. In this environment, developers (and particularly more senior developers) are trusted to build whatever they think has the most value, often by necessity. Some people will struggle a lot in this environment (particularly those people who are very uncomfortable being self-directed); others will waste some efforts, but good and proactive developers will produce a lot of stuff, in a plethora of directions. They will also explore, optimize, experiment with different approaches which may not pan out, etc., all virtually unconstrained by process and overhead. In this environment, it's not uncommon to have 10x productivity from each person, and only half of that work actually end up being used in production (compared to the carefully controlled work product in larger companies).

But, small companies get some side-benefits from that environment also, in addition to the overall increases in per-person effective productivity. Because developers are experimenting and less constrained by processes and other people's priorities, they will often build things which would never have been conceived of within meetings among managers and PM's. Often these are "hidden" things (code optimizations, refactoring to reduce maintenance costs, process optimizations for developer workflows, "fun" feature adds, etc.), but sometimes they are "interesting" things, of the sort which could be construed as innovations. It is these developments which will typically give rise to actual advances in product areas, and ultimately lead to business value through patents which have meaning in the markets.

Now, I'd be remiss to not mention that a number of companies are aware of this fact, and have done things to try to mitigate these effects. Google's famous "20% time", for example, was almost certainly an attempt to address this head-on, by creating an internal environment where innovation was still possible even as the company grew (note: they eventually got too large to sustain this in the face of profit desires from the market). Some companies use hackathons for this, some have specific groups or positions which are explicitly given this trust and freedom, etc. But by and large, they are all just trying to replicate what tends to happen organically at smaller companies, which do not have the pressure or resources build all the systems and overhead to get in the way of their own would-be success.

Anyway, hopefully that's somewhat insightful as to why most real innovation happens in smaller companies, at least in the software industry.

How not to do Agile

Note: This post is intended to be a tongue-in-cheek take, based on an amalgam of experiences and anecdotes, and is not necessarily representative of any specific organization. That said, if your org does one or more of these things, it might be beneficial to examine if those practices are really beneficial to the org or not.

The following is a collection of things you should not do when trying to do Agile, imho; these practices either run counter to the spirit of the methodology, will likely impede realizing the value from doing such, and/or demonstrate a fundamental misunderstanding of the concept(s).

Mandating practices top-down

One of the core precepts of Agile is that it is a "bottom-up" organization system, which is intended to allow developers to tweak the process over time to optimize their own results. Moreover, it is very important in terms of buy-in for developers to feel like the process is serving the needs of development first and foremost. When mandated from outside of development, even an otherwise optimal process might not get enough support over time to be optimally adopted and followed.

It is very often a sign of "Agile in name only" within organizations when this is mandated by management, rather than adopted organically (and/or with buy-in across the development teams). This is one of the clearest signals that an organization either has not bought into Agile, and/or the management has a fundamental misunderstanding of what Agile is.

Making your process the basis of work

One of the tenants of Agile is that process is there in service of the work product, and should never be the focus for efforts. As such, if tasks tend to revolve around process, and/or are dependent on specific process actions, this should be considered a red flag. Moreover, if developers are spending a non-trivial amount of time on process-related tasks, this is another red flag: in Agile, developers should be spending almost all their time doing productive work, not dealing with process overhead.

One sign that this might be the case is if/when workflows are heavily dependent on the specifics of the process and/or tooling, as opposed to the logical steps involved in getting a change done. For example, if a workflow starts with "first, create a ticket...", this is not Agile (at least in spirit, and probably in fact). If the workflow is not expressed in terminology which is process and tooling independent, the org probably isn't doing Agile.

Tediously planning future release schedules

Many organizations with a Waterfall mindset always plan out future releases, inclusive of which changes will be included, what is approved, what is deferred, etc. This (of course) misses the point of Agile entirely, since (as encapsulated in the concept of Agile) you cannot predict the timeline for changes of substance, and this mentality makes you unable to adopt to changing circumstances and/or opportunities (ie: be "agile"). If your organization is planning releases with any more specificity than a general idea of what's planned for the next release, and/or it would be a non-trivial effort to include changes of opportunity into a release, then the org isn't doing Agile.

Gating every software change

The Agile methodology is inherently associated with the concept of Continuous Improvement, and although the two can be separated conceptually, it's hard to imagine an Agile environment which did not also emphasize CI. Consequently, in an Agile environment, small incremental improvements are virtually always encouraged, both explicitly via process ideals, and implicitly via low barriers. Low barriers is, in fact, a hallmark of organizations with high code velocity, and effectively all high productive Agile dev teams.

Conversely, if an organization has high barriers in practice to code changes (process wise or otherwise), and/or requires tedious approvals for any changes, this is a fairly obvious failure in terms of being Agile. Moreover, it's probably a sign that the organization is on the decline in general, as projects and teams where this is the prevailing mentality and/or process tend to be fairly slow and stagnant, and usually in "maintenance mode". If this doesn't align with management's expectations for the development for a project, then the management might be poor.

Create large deferred aggregate changes

One of the precepts of Agile is biasing to small, incremental changes, which can be early integrated and tested as self-contained units. Obviously, large deferred aggregate changes are the antithesis of this. If your organization has a process which encourages or forces changes to be deferred and/or grow in isolation, you're certainly not doing Agile, and might be creating an excessive amount of wasteful overhead also.

Adding overhead in the name of "perfection"

No software is perfect, but that doesn't stop bad/ignorant managers from deluding themselves with the belief that by adding enough process overhead, they can impose perfection upon a team. Well functioning Agile teams buck this trend through self-organization and control of their own process, where more intelligent developers can veto these counter-productive initiatives from corporate management. If you find that a team is regularly adding more process to try to "eliminate problems", that's not only not Agile, but you're probably dealing with some bad management as well.

Have bad management

As alluded to in a number of the points above, the management for a project/team has a huge impact on the overall effectiveness of the strategies and processes. Often managers are the only individuals who are effectively empowered to change a process, unless that ability is clearly delegated to the team itself (as it would be in a real Agile environment). In addition to this, though, managers shape the overall processes and mindsets, in terms of how they manage teams, what behaviors are rewarded or punished, how proactively and clearly they articulate plans and areas of responsibility, etc. Managers cannot unilaterally make a team and/or process function well, but they can absolutely make a team and/or process function poorly.

Additionally, in most organizations, manager end up being ultimately responsible for the overall success of a project and/or team, particularly when in a decision making role, because they are the only individuals empowered to make (or override) critical decisions. A good manager will understand this responsibility, and work diligently to delegate decisions to the people most capable of making them well, while being proactively vigilant for intrusive productivity killers (such as heavy process and additional overhead). Conversely, a bad manager either makes bad decisions themselves, or effectively abdicates this responsibility through inaction and/or ignorance, and allows bad things to happen within the project/team without acknowledging responsibility for those events. If the project manager doesn't feel that they are personally responsible for the success of the product (perhaps in addition to others who also feel that way), then that manager is probably incompetent in their role, and that project is likely doomed to failure in the long run unless they are replaced.

Take home

Agile is just one methodology for software development; there are others, and anything can work to various degrees. However, if you find yourself in a position where the organization claims to be "agile", but exhibits one or more of the above tendencies, know that you're not really in an organization which is practicing Agile, and their self-delusion might be a point of concern. On the other hand, if you're in a position to influence and/or dictate the development methodology, and you want to do Agile, make sure you're not adding or preserving the above at the same time, lest you be the one propagating the self-delusion. Pick something that works best for you and your team, but make an informed choice, and be aware of the trade-offs.