Some thoughts on code reviews

Code reviews (https://about.gitlab.com/topics/version-control/what-is-code-review/) are a fairly standard practice in the industry, especially within larger companies. The process of having multiple developers look at code changes critically is found in several development methodologies (eg: extreme programming, paired programming, etc.), and they are often perceived as essentially for maintaining a level of overall code quality. I'd imagine that no respectable engineering leader in any larger organization would accept a process which did not incorporate mandatory code reviews in some form.

So with that intro, here's a bit of a hot/controversial take: I think code reviews are overrated.

Before I dive into why I think this, a quick tangent for an admission: the majority of code I've actually written in my career, personally and professionally, has been done without a formal code review process in place (and thus, not code reviewed). I've also, personally, experienced considerably more contention and strife generated by code reviews than value-add. So I certainly have a perspective here which is biased by my own experience... but I also don't think my experience is that unique, and/or would be that dissimilar to the experience of many other people, based on conversations I've had.

So that being said, why do I not subscribe to the common wisdom here?

I know a manager, who shall remain unnamed here, for whom the solution for every problem (bug, defect, customer issue, design oversight, etc.) is always one of two things: we should have caught that with more code review, or we should have caught that with more unit testing (or both). His take represents the sort of borderline brainless naivety which is all too common among nominally technical managers who have never accomplished anything of significance in their careers, and have managed to leverage their incompetence into high-paying positions where they parrot conventional wisdom and congratulate themselves, while contributing no positive value whatsoever to their employing organizations.

The common perception of this type of manager (and I have known several broadly with this mindset) is that any potential product failure can be solved by more process, and/or more strict adherence to a process. There is not a problem they have ever encountered, in my estimation, for which their would-be answer is not either adding more process, or blaming the issue on the failure of subordinates to follow the process well enough. To these people, if developers just stare at code long enough, it will have no bugs whatsoever, and if code which passes a code review has a bug, it was because the reviewers didn't do a good enough job, and should not have approved it.

Aside: The above might sound absurd to anyone who has spent any time working in the real world, but I assure you that it is not. I know more than one manager who's position is that no code should ever be approved in a code review, and/or be merged into the company's repository, which has any bugs whatsoever, and if there are any bugs in the code, the code review approver(s) should be held accountable. I think this is borderline malicious incompetence, but some of these people have failed upward into positions where they have significant power within organizations, and this is absolutely a very real thing.

Back to code reviews, though: I think they are overrated based on a couple perceptions:

• The most important factor in producing high-value code over time is velocity (of development), hands down
• Code reviews rarely catch structural design issues (and even when they do, by that time, it's often effectively too late to fix them)
• Code reviews encourage internal strife from opinionated feedback, which often has little value on overall code quality
• Code reviews often heavily implicitly bias against people without as many social connections, and/or those which do not "play politics" within the org/teams (and conversely, favor those that do, encouraging that behavior)
• As per above, code reviews are very often abused by bad managers, which can easily lead to worse overall outcomes for orgs

To be clear and fair, code reviews have some tangible benefits, and I wouldn't necessarily dispose of them all together, were I running a dev org. In particular, the potential benefits such as sharing domain knowledge, increasing collaboration, and propagating best-practices (particularly when mentoring more junior developers) are tangible benefits of code reviews or equivalent. There is a reasonably compelling argument that, with good management in place, and when not used for gating and/or misused for blame attribution, code reviews have sufficient positive value to be a good practice.

However, the risks here are real and substantial, and this is not something which is a clear win in all, or perhaps even most, cases. Code reviews impact velocity, and the positive value proposition must be reasonably high for them to have a net positive value, given that. You're not likely to catch many actual bugs in code reviews, and if your developers use them as a crutch for this (psychologically or otherwise), that's another risk. If you have management which thinks thorough enough code reviews will give you "pristine" code, you're almost certainly better off eliminating them entirely (in concept), in my estimation (assuming you cannot replace those terrible managers). Code reviews are something which can have net positive value when used appropriately... but using them appropriately is something I've personally seen far less often than not.

That's my 2c on the topic, anyway.

Development process issues in different types of orgs

There's an interesting dichotomy I've observed between companies where the development is run by developers, and those where it's run by managers.

In the former case, problems tend to get fixed, because the people making decisions are also the people impacted by the problems.

In the latter case, problems tend to compound and go unresolved, because the people making decisions are not directly affected, and/or cannot understand why the situation is problematic, and/or have competing priorities (to fixing them). This creates a situation where you really don't want to be doing development, and the best developers tend to move into management, and/or onto different projects.

The latter is also the case where projects tend to die over time, and sometimes companies with them, as they eventually become unmaintainable.

All of the above might be obvious to others, but I don't have a lot of previous experience working on teams where the development methodologies are all being dictated by management (most of my previous roles have been in companies using Agile, or something close to it, and/or smaller companies). In Agile ish organizations, developers were motivated and empowered to fix problems which impeded shipping code, and as such when there are blockers, they tend to be short-lived (sometimes the resolutions are not optimal, but significant issues rarely linger unresolved). This is not the case in large orgs where the methodologies are dictated my managers, though, particularly when those managers are not actively working on code.

This needs to be added to my list of questions and concerns when considering future roles, I think.

Thoughts on recent Google layoffs, Python support team

Recently, Google laid off an internal group responsible for maintaining Python within the company (story: https://www.hindustantimes.com/business/google-layoffs-sundar-pichai-led-company-fires-entire-python-team-for-cheaper-labour-101714379453603.html). Nominally, this was done to reduce costs; purportedly they will look to hire replacement people for the group in Germany instead, which will be cheaper. This is what the previous team was nominally responsible for: https://news.ycombinator.com/item?id=40176338

Whether or not this saves Google money in the longer term, on balance, is an open question. This doesn't normally work out well (getting rid of tribal knowledge, experience, etc.), but this is something large companies do regularly, so not a huge surprise in general. But this isn't what was most interesting about this to me.

Rather, I'd like to focus on some tidbits in the reporting and personal accounts which reveal some interesting (if true) things about the inner workings at Google at this moment.

Python is deprecated within Google

According to one of the affected developers, Python is considered tech debt within Google, with existing code to be replaced with code in other languages, and new code in Python frowned upon. There are various reasons given for this, but the fact that Google is moving away from Python is interesting, when this doesn't seem to be the case in general in the industry (and Google often is ahead of the curve with technology migrations).

The group was responsible for fixing all impacted code, even other groups'

When the group upgraded Python across Google, they needed to fix all impacted code, even if they didn't own it. This is a pretty huge headwind to updating versions and taking patches, and is reflected in their update planning and schedules. This points to the rather large systemic problem of trying to take regular updates to libraries across a large code base, and either a lack of planning, or a lack of budgeting at the project level(s).

Related to this, the group noted that often unit tests would be flaky, because they were written in a fragile way, and broke with updates to the language version. This is the large systemic problem with having a large code base of unit tests, of course: you need to have a plan and resource allocation for maintaining them over time, for them to have net positive value. It seems like Google perhaps is lacking in this area also.

Companies often don't value infrastructure teams

This is a general issue, but something to watch out for when charting a career course: lots of companies undervalue and under-appreciate infrastructure teams, who act as effectively force multiplier (in best cases) for product teams. The large the org, the less visibility the people working on foundational structures have, and the more common it is for upper management to look at those teams for cost cutting. Getting into foundational work within a larger org is more risky, career-wise: it might be the most effective value-add for the company, but it's also likely to be the least appreciated as well.

If you want to maximize your career potential, flashy prototype projects supported by mountains of tech debt which you can hand off and move on to the next flashy project will get you the most positive recognition at almost all large companies. A close second-place is the person who fixes high-visibility issues with kludgy "fixes" which seem to work, but ignore underlying or systemic problems (which can be blamed on someone else). It's extremely probable both of those types of developers will be more highly valued than someone who builds and maintains non-flashy but critical support infrastructure.

Managers are generally dumb, don't understand actual impacts

This is somewhat of a corollary to the above, but the person deciding which group to downsize to ensure profits beat expectations and the executives get their multi-million dollar performance bonuses isn't going to have any idea what value people in lower-level groups actually bring, and/or what might be at risk by letting critical tribal knowledge walk out the door. When there need to be cuts (usually for profit margins), the most visible projects (to upper management) will be the ones where people are most safe. Don't get complacent and think that just because your project is critical to the company, and/or your value contribution is high, that your job is more safe. Pay attention to what executives talk about at all-hands meetings: the people building prototype features in those groups are the people who are most valued to the people making the layoff decisions.

Take home

While Google has declined substantially since it's heyday (in terms of prestige and capability), in some ways it is still a bellwether for the industry, so it's good to pay attention to what goes on there. In this case, I think there's good information to be gleaned, beyond just the headline info. It sounds like Google is now more similar than dissimilar to a large tech company on the decline, though.

The problem of "thrashing"

"Thrashing" is a general term/issue in computer science, which refers to the situation (in the abstract) in which multiple "work items" are competing for the same set of resources, and each work item is being processed in chunks (ie: either in parallel, or interleaved), and as a result the resource access ping-pongs between the different work items. This can be very inefficient for the system if switching access to the resources causes overhead. Here's the wiki page on the topic: https://en.wikipedia.org/wiki/Thrashing_(computer_science)

There are numerous examples of thrashing issues in software development, such as virtual memory page faults, cache access, etc. There is also thread context thrashing, where when you have too many threads competing for CPU time, the overhead of just doing thread context switching (which is generally only a few hundred CPU cycles) can still overwhelm the system. When thrashing occurs, it is generally observed as a non-linear increase in latency/processing time, relative to the work input (ie: the latency graph "hockey sticks"). At that point, the system is in a particularly bad state (and, ironically, a very common critical problem in orgs is that additional diagnostic processes get triggered to run in that state, based on performance metrics, which can then cause systems to fail entirely).

To reduce thrashing, you generally want to try to do a few things:

• Reduce the amount of pending parallel/interleaved work items on the system
• Allocate work items with more locality if possible (to prevent thrashing relative to one processing unit, for example)
• Try to allow more discrete work items to complete (eg: running them longer without switching), to reduce the context switching overhead

Now, while all of the above is well-known in the industry, I'd like to suggest something related, but which is perhaps not as well appreciated: the same problems can and do occur with respect to people within an organization and process.

People, as it turns out, are also susceptible to some amount of overhead when working on multiple things, and task switching between them. Moreover, unlike computers, there is also some overhead for work items which are "in flight" for people (where they need to consider and/or refresh those items just to maintain the status quo). The more tasks someone is working on, and the more long-lived work items are in flight at any given time, the move overhead exists for that person to manage those items.

In "simple" jobs, this is kept minimal on purpose: a rote worker might have a single assigned task, or a checklist, so they can focus on making optimal progress on the singular task, with the minimal amount of overhead. In more complex organizations, there are usually efforts to compartmentalize and specialize work, such that individual people do not need to balance more than an "acceptable" number of tasks and responsibilities, and to minimize thrashing. However, notably, there are some anti-patterns, specific to development, which can exacerbate this issue.

Some notable examples of things which can contribute to "thrashing", from a dev perspective:

• Initiatives which take a long time to complete, especially where other things are happening in parallel
• Excessive process around code changes, where the code change can "linger" in the process for a while
• Long-lived branches, where code changes needs to be updated and refreshed over time
• Slow pull-request approval times (since each outstanding pull-request is another in-progress work item, which requires overhead for context switching)
• Excessive "background" organizational tasks (eg: email management, corporate overhead, Slack threads, managing-up tasks, reporting overhead, side-initiatives, etc.)

Note, also, that there is a human cost to thrashing as well, as people want to both be productive and see their work have positive impacts, and thrashing hurts both of these. As a manager, you should be tracking the amount of overhead and "thrashing" that your reports are experiencing, and doing what you can to minimize this. As a developer, you should be wary of processes (and potentially organizations) where there are systems in place (or proposed) which contribute to the amount of thrashing which is likely to happen while working on tasks, because this has a non-trivial cost, and the potential to "hockey stick" the graph of time wasted dealing with overhead.

In short: thrashing is bad, and it's not just an issue which affects computer systems. Not paying attention to this within an org can have very bad consequences.

The problem with process

Note: This post might more accurately be titled "one problem with process", but I thought the singular had more impact, so there's a little literary license taken. Also, while this post is somewhat inspired by some of my work experiences, it does not reflect any particular person or company, but rather a hypothetical generalized amalgamation.

There's an adage within management, which states that process is a tool which makes results repeatable. The unpacking of that sentiment is that if you achieve success once, it might be a fluke, dependent on timing or the environment, dependent on specific people, etc., but if you have a process which works, you can repeat it mechanically, and achieve success repeatedly and predictably. This is the mental framework within which managers add process to every facet of business over time, hoping to "automate success".

Sometime it works, sometimes it doesn't. Often process is also used to automate against failure as well, by automating processes which avoid perceived and/or historical breakdown points. This, more often than not, is where there be landmines.

Imagine a hypothetical: you're a manager, grappling with a typical problem of quality and execution efficiency. You want to increase the former, without sacrificing the latter (and ideally, with increasing the latter as well). Quality problems, as you know, come from rushing things into production without enough checks and sign-offs; process can fill that gap easily. But you also know that with enough well-defined process, people become more interchangeable in their work product, and can seamlessly transition between projects, allowing you to optimally allocate resources (in man-months), and increase overall execution efficiency.

So you add process: standard workflows for processing bugs, fields in the tracking system for all the metrics you want to measure, a detailed workflow that captures every state of every work item that is being worked, a formalized review process for every change, sign-offs at multiple levels, etc. You ensure that there is enough information populated in your systems such that any person can take over any issue at any time, and you'll have full visibility into the state of your org's work at all times. Then you measure your metrics, but something is wrong: efficiency hasn't increased (which was expected, it will take time for people to adjust to the new workflows and input all the required data into the systems), but quality hasn't increased either. Clearly something is still amiss.

So you add more process: more stringent and comprehensive testing requirements, automated and manual, at least two developers and one manager reviewing every change which goes into the code repository, formalized test plans which must be submitted and attested to along with change requests, more fields to indicate responsible parties at each stage, more automated static analysis tools, etc. To ensure that the processes are followed, you demand accountability, tying sign-off for various stages to performance metrics for responsible employees. Then you sit back and watch, sure that this new process is sufficient to guarantee positive results.

And yet... still no measurable improvement in overall perceived product quality. Worse, morale is declining: many employees feel stifled by the new requirements (as they should; those employees were probably writing the bugs before), they are spending large amounts of time populating the process data, and it's taking longer to get fixes out. This, in turn, is affecting customers satisfaction; you try to assure them that the increased quality will compensate for the longer lead times, but privately your metrics do not actually support this either. The increased execution efficiency is still fleeting as well: all the data is there to move people between project seamlessly, but for some reason people still suffer a productivity hit when transitioned.

Clearly what you need is more training and expertise, so you hire a Scrum master, and contract for some Scrum training classes. Unsure where everyone's time is actually going, you insist that people document their work time down to 10 minute intervals, associating each block of time with the applicable ticket, so that time can be tracked and optimized in the metrics. You create tickets for everything: breaks, docs, context switches, the works. You tell your underling managers to scrutinize the time records, and find out where you are losing efficiency, and where you need more process. You scour the metrics, hoping that the next required field will be the one which identifies the elusive missing link between the process and the still lacking quality improvements.

This cycle continues, until something breaks: the people, the company, or the process. Usually it's one of the first two.

In the aftermath, someone asks what happened. Process, metrics, KPI's: these were the panaceas which were supposed to lead to the nirvana of efficient execution and high quality, but paradoxically, the more that were added, the more those goals seemed to suffer. Why?

Aside: If you know the answer, you're probably smarter than almost all managers in most large companies, as the above pattern is what I've seen (to some degree) everywhere. Below I'll give my take, but it is by no means "the answer", just an opinion.

The core problem with the above, imho, is that there is a misunderstanding of what leads to quality and efficiency. Quality, as it turns out, comes from good patterns and practices, not gating and process. Good patterns and practices can come from socializing that information (from people who have the knowledge), but more often than not come from practice, and learned lessons. The quantity of practice and learned lessons come from velocity, which is the missing link above.

Process is overhead: it slows velocity, and decreases your ability to improve. Some process can be good, but only when the value to the implementers exceeds the cost. This is the second major problem in the above hypothetical: adding process for value of the overseers is rarely if ever beneficial. If the people doing the work don't think the process has value to them, then it almost certainly has net negative value to the organization. Overseers are overhead; their value is only realized if they can increase the velocity of the people doing the work, and adding process rarely does this.

Velocity has another benefit too: it also increases perceived quality and efficiency. The former happens because all software has bugs, but customers perceive how many bugs escape to production, and how quickly they are fixed. By increasing velocity, you can achieve pattern improvement (aka: continuous improvement) in the code quality itself. This decreases the number of overall issues as a side-effect of the continuous improvement process (both in code, and in culture), with a net benefit which generally exceeds any level of gating, without any related overhead. If you have enough velocity, you can even also increase test coverage automation, for "free".

You're also creating en environment of learning and improvement, lower overhead, less restrictions, and more drive to build good products among your employees who build things. That tends to increase morale and retention, so when you have an issue, you are more likely to still have the requisite tribal knowledge to quickly address it. This is, of course, a facet of the well-documented problem with considering skill/knowledge workers in terms of interchangeable resource units.

Velocity is the missing link: being quick, with low overhead, and easily pivoting to what was important without trying to formalize and/or add process to everything. There was even a movement a while ago which captured at least some of the ideals fairly well, I thought: it was called Agile Development. It seems like a forgotten ideal in the environments of PKI's, metrics, and top-heavy process, but it's still around, at least in some corners of the professional world. If only it didn't virtually always get lost with "scale", formalization, and adding "required" process on top of it.

Anyway, all that is a bit of rambling, with which I hope to leave the reader with this: if you find yourself in a position where you have an issue with quality and/or efficiency, and you feel inclined to add more process to improve those outcomes, consider carefully if that will be the likely actual outcome (and as necessary, phone a friend). Your org might thank you eventually.

 

Some thoughts on budget product development, outsourcing

I've been thinking a bit about the pros and cons of budget/outsourcing product development in general. By this, I mean two things, broadly: either literally outsourcing to another org/group, or conducting development in regions where labor is cheaper than where your main development would be conducted (the latter being, presumably, where your main talent and expertise resides). These are largely equivalent in my mind and experience, so I'm lumping them together for purposes of this topic.

The discussion has been top-of-mind recently, for a few reasons. One of the main "headline" reasons is all the issues that Boeing is having with their airplanes; Last Week Tonight had a good episode about how aggressive cost-cutting efforts have led to the current situation there, where inevitable quality control issues are hurting the company now (see: https://www.youtube.com/watch?v=Q8oCilY4szc). The other side of this same coin, which is perhaps more pertinent to me professionally, is the proliferation of LLM's to generate code (aka: "AI agents"), which many people think will displace traditional more highly-compensated human software developers. I don't know how much of a disruption to the industry this will eventually be, but I do have some thoughts on the trade-offs of employing cheaper labor to an organization's product development.

Generally, companies can "outsource" any aspect of product development, and this has been an accessible practice for some time. This is very common in various industries, especially for so-called "commoditized" components; for example, the automobile industry has an entire sub-industry for producing all the various components which are assembled into automobiles, and usually acquired from the cheapest vendors. This is generally possible for any components which are not bespoke, across any industry with components which are standardized, and can be assembled into larger products.

Note that this is broadly true in the software context as well: vendors sell libraries with functionality, open source libraries are commonly aggregated into products, and component re-use is fairly common in many aspects of development. This can even be a best-practice in many cases, if the component library is considered near the highest quality and most robust implementation of functionality (see: the standard library in C++, for example). Using a robust library which is well-tested across various usage instances can be a very good strategy.

Unfortunately, this is less true in the hardware component industries, since high-quality hardware typically costs more (in materials and production costs), so it's generally less feasible to use the highest quality components from a cost perspective. There is a parallel in first-party product development, where your expected highest quality components will usually cost more (due to the higher costs for the people who produce the highest quality components). Thus, most businesses make trade-offs between quality and costs, and where quality is not a priority, tend to outsource.

The danger arises when companies start to lose track of this trade-off, and/or misunderstand the trade-offs they are making, and/or sacrifice longer-term product viability for short-term gains. Each of these can be problematic for a company, and each are inherent dangers in outsourcing parts of development. I'll expand on each.

Losing track of the trade-offs is when management is aware of the trade-offs when starting to outsource, but over time these become lost in the details and constant pressure to improve profit margins, etc. For example, a company might outsource a quick prototype, then be under market pressure to keep iterating on it, while losing track of (and not accounting for) the inherent tech debt associated with the lower quality component. This can also happen when the people tracking products and components leave, and new people are hired without knowledge of the previous trade-offs. This is dangerous, but generally manageable.

Worse that the above is when management doesn't understand the trade-offs they are making. Of course, this is obviously indicative of poor and incompetent management, yet time and time again companies outsource components without properly accounting for the higher long-term costs of maintaining and enhancing those components, and companies suffer as a result. Boeing falls into this category: by all accounts their management thought they could save costs and increase profits by outsourcing component production, without accounting for the increased costs of integration and QA (which would normally imply higher overall costs for any shipping and/or supported product). That's almost always just egregious incompetence on the part of the company's management, of course.

The last point is also on display at Boeing: sacrificing long-term viability for short-term gains. While it's unlikely this was the motivation in Boeing's case, it's certainly a common MO with private equity company ownership (for example) to squeeze out as much money as possible in the short term, while leaving the next owners "holding the bag" for tech debt and such from those actions. Again, this is not inherently bad, not every company does this, etc.; this is just one way companies can get into trouble, by using cheaper labor for their product development.

This bring me, in a roundabout way, to the topic of using LLM's to generate code, and "outsource" software product development to these agents. I think, in the short term, this will pose a substantial risk to the industry in general: just as executives in large companies fell in love with offshoring software development in the early 2000's, I think many of the same executives will look to reduce costs by outsourcing their expensive software development to LLM's as well. This will inevitably have the same outcomes over the long run: companies which do this, and do not properly account for the costs and trade-offs (as per above), will suffer, and some may fail as a results (it's unlikely blame will be properly assigned in these cases, but when companies fail, it's almost always due to bad executive management decisions).

That said, there's certainly also a place for LLM code generation in a workflow. Generally, any task which you would trust to an intern, for example, could probably be completed by a LLM, and get the same quality of results. There are some advantages to using interns (eg: training someone who might get better, lateral thinking, the ability to ask clarifying questions, etc.), but LLM's may be more cost effective. However, if companies largely stop doing on-the-job training at scale, this could pose some challenges for the industry longer-term, and ultimately drive costs higher. Keep in mind: generally, LLM's are only as "good" as the sum total of average information online (aka: the training data), and this will also decline over time as LLM output pollutes the training data set as well.

One could argue that outsourcing is almost always bad (in the above context), but I don't think that's accurate. In particular, outsourcing, and the pursuit of short-term profits over quality, does serve at least two valuable purposes in the broader industry: it helps new companies get to market with prototypes quickly (even if these ultimately need to be replaced with quality alternatives), and it helps older top-heavy companies die out, so they can be replaced by newer companies with better products, as their fundamentally stupid executives make dumb decisions in the name of chasing profit margins (falling into one of more of the traps detailed above). These are both necessary market factors, which help industries evolve and improve over time.

So the next some some executive talks about outsourcing some aspect of product development, either to somewhere with cheaper labor or to a LLM (for example), you can take some solace in the fact that they are probably helping contribute to the corporate circle of life (through self-inflicted harm), and for each stupid executive making stupid decisions, there's probably another entrepreneur at a smaller company who better understands the trade-offs of cheaper labor, is looking to make the larger company obsolete, and will be looking for quality product development. I don't think that overall need is going to vanish any time soon, even if various players shuffle around.

My 2c, anyway.

Mobile devices and security

Generally, passwords are a better form of security than biometrics. There are a few well-known reasons for this: passwords can be changed, cannot be clandestinely observed, are harder to fake, and cannot be taken from someone unwillingly (eg: via government force, although one could quibble about extortion as a viable mechanism for such). A good password, used for access to a well-designed secure system, is probably the best known single factor for secure access in the world at present (with multi-factor including a password as the "gold standard").

Unfortunately, entering complex passwords is generally arduous and tedious, and doubly so on mobile devices. And yet, I tend to prefer using a mobile device for accessing most secure sites and systems, with that preference generally only increasing as the nominal security requirements increase. That seems counter-intuitive at first glance, but in this case the devil is in the details.

I value "smart security"; that is, security which is deployed in such a way as to increase protection, while minimizing the negative impact on the user experience, and where the additional friction from the security is proportional to the value of the data being protected. For example, I use complex and unique passwords for sites which store data which I consider valuable (financial institutions, sensitive PII aggregation sites, etc.), and I tend to re-use password on sites which either don't have valuable information, or where I believe the security practices there to be suspect (eg: if they do something to demonstrate a fundamental ignorance and/or stupidity with respect to security, such as requiring secondary passwords based on easily knowable data, aka "security questions"). I don't mind entering my complex passwords when the entry is used judiciously, to guard against sensitive actions, and the app/site is otherwise respectful of the potential annoyance factor.

Conversely, I get aggravated with apps and sites which do stupid things which do nothing to raise the bar for security, but constantly annoy users with security checks and policies. Things like time-based password expiration, time-based authentication expiration (especially with short timeouts), repeated password entry (which trains users to type in passwords without thinking about the context), authentication workflows where the data flow is not easily discernible (looking at most OAuth implementations here), etc. demonstrate either an ignorance of what constitutes "net good" security, or a contempt for the user experience, or both. These types of apps and sites are degrading the security experience, and ultimately negatively impacting security for everyone.

Mobile OS's help mitigate this, somewhat, by providing built-in mechanisms to downgrade the authentication systems from password to biometrics in many cases, and thus help compensate for the often otherwise miserable user experience being propagated by the "security stupid" apps and sites. By caching passwords on the devices, and allowing biometric authentication to populate them into forms, the mobile devices are "downgrading" the app/site security to single factor (ie: the device), but generally upgrading the user experience (because although biometrics are not as secure, they are generally "easy"). Thus, by using a mobile device to access an app/site with poor fundamental security design, the downsides can largely be mitigated, at the expense of nominal security in general. This is a trade-off I'm generally willing to make, and I suspect I'm not alone in this regard.

The ideal, of course, would be to raise the bar for security design for apps and sites in general, such that security was based on risk criteria and heuristics, and not (for example) based on arbitrary time-based re-auth checks. Unfortunately, though, there are many dumb organizations in the world, and lots of these types of decisions are ultimately motivated or made by people who are unable or unwilling to consider the net security impact of their bad policies, and/or blocked from making better systems. Most organizations today are "dumb" in this respect, and this is compounded by standards which mandate a level of nominal security (eg: time-based authentication expiration) which make "good" security effectively impossible, even for otherwise knowledgeable organizations. Thus, people will continue to downgrade the nominal security in the world, to mitigate these bad policy decisions, with the tacit acceptance from the industry that this is the best we can do, within the limitations imposed by the business reality in decision making.

It's a messy world; we just do the best we can within it.