As a programmer, uncertainty is my second greatest obstacle. Second to my ability to fit large problems in my head. Uncertainty comes from not being able to fit the entire problem in my head and being able to compare the different ways problem can be solved.

With experience comes the ability to assign create new kinds of heuristics for various problems, which makes it easier to fit them in my head as I work. Reasoning about them, given reasonable simplifications, require experience. Lots of it.

And while acquiring experience you make mistakes, costly ones. You might put lots of time into it without really being able to solve the right problem. It can make you a bit scared of trying it next time. So you avoid it. I think some programmers become so scared of 'doing it the right' because of the risk associated with it that they avoid it more or less all together.

The good programmers are those who manage to get enough experience to reduce the risk before getting scared off by their mistakes.

That's the theory I just came up with anyway.

You should never have to "fit a large problem" into your head, at least in detail, at once. I think it's something of a myth that that's what great programmers do, when in fact they structure their code to avoid it.

I've built immensely complex software and in the end the code was good not because I was smart, but because I am stupid, and adapted the code to work around my limitations. It's the "smart" programmers I worry about :)

The key to conquering complex problems is to layer it into levels of abstractions, dividing it into ever smaller parts that does one thing. Then one step up in the abstraction you don't have to worry about the details beneath. If you think about it that's how programming languages themselves work, we no longer push around registers and instructions, its turtles all the way down.

Your mind should always be able to encompass the level its currently on. That's why reducing the amount of code is important. Increasing the entropy in your code makes it more readable and understandable. It also makes it easier to spot patterns. Whenever your code starts feeling beyond easy comprehension it's time to divide it further.

If a problem feels to large, my advice is just to start doing classes or methods that attacks different aspects of the problem, and then compose those.

There's a nice quote by Linus:

"Bad programmers worry about the code. Good programmers worry about data structures and their relationships."

Good structures will shield you from a lot of complexities and simplify your code. Structures are easy to test. Always consider if there isn't a collection or other data structure that could swallow some complexity.

Also, never be afraid of making mistakes, coding is an explorative process and the most valuable thing you can do is to do a mistake and realize it, because that also often means the solution presents itself clearly.

No code is wasted code because even though you might end up deleting all of it the lessons they leave behind are valuable.

In the end coding is a lot like writing a book, get everything down on paper/code without worrying too much and then iterate and move things around until its good :)

I don't personally use functional programming languages on a daily basis, but this, I feel, is the veil between programmers who are highly successful and those who limp along: being able to have a mathematical (side effect free) approach to solving problems. This requires being able to break a larger problem into subsets. The closer you can get your methods to not having affects on the global scope, the better a programmer you will become. In other words, you don't have to be able to keep the whole problem in you head, but you have to be able to recognize how to subdivide the problem into manageable bits.

The other major trap that I think affects programmers is not appreciating what is happening when you transform your data: when you make a change to data state, how does that affect the system? Where are you persisting that change? Do you need to persist a change or is it simply a temporal snapshot of something happening?

Basically if somebody claims to be a "good" programmer, but they don't have a solid grasp on set theory, they I'm sceptical that they will be a "good" programmer in the long run. Naturally you also have to become familiar with the language you are using and the environment you are developing for.

I'm not going to argue that refactoring is unnecessary, far from it. However with each refactor there is always the possibility of introducing a hard to debug regression. Even if you follow rigorous TDD, there is no such thing as 100% code coverage with your tests which means that new bugs can and do creep in.

I guess what I'm saying is that refactoring shouldn't be entered into lightly.

I agree with you to a certain extent but... the road to code hell is paved with "don't disturb the tests" intentions ;)

This is a bigger problem the less continuously you refactor. I've seen this mentality in a lot of projects, where they don't dare "mess with the code" because it might introduce bugs or break the tests... but IMHO this is where I think TDD sometimes goes too far.

It should ring warning bells if your code, tests and development cycle is so fragile that you can't do continuous cleanup. In its extreme this is the lava flow anti-pattern.

It must always be preferable to get the code base as compact and simple as possible but bugs will be easier to find and the code easier to understand for everyone... otherwise you're just accumulating technical debt.

Rather than storing nitroglycerin in a pillow factory it usually is a good idea to make it less volatile.

If you refactor in a way that you improve code readability and reduce complexity, you substantially reduce the likelihood of having an undiscovered bug that is already in your code.

I've found the opposite was a worse problem - programmers who discovered some new "pattern" and would rewrite the whole codebase every week. Endless churn and zero business value.

Code generation and config files are evil. If it affects program behaviour, it belongs in the codebase, where you can search for it, and where it's part of the normal release/review/testing cycle.

And code complete, clean code...

That's like writing a book "correctly" the first time without drafting (there are a few authors that does that, but not many). Or drawing a picture without sketching.

Much of the coding process is gaining knowledge of the problem you're trying to solve and discovering patterns as you go along.

Also typically features are tacked on over time, and it reaches a point where new patterns are needed that wasn't needed before.

I agree with you in some sense though, some patterns are low-cost and high-reward in versatility & maintainability. Like using events rather than explicit references, and you should use them from the start as a matter of "good habits".

Some more overreaching patterns like factories, modules, plug-ins etc etc only make sense when the code reaches a certain scope.

William Deming famously said that when focus is on quality, quality rises and costs fall. And that when focus is cost, costs rise and quality falls.

We live in a time where we're encouraged to make mistakes. Refactoring, fail fast and often... I get the impression sometimes that we can go on failing forever and its ok. Everything is awesome. Because its fun writing code right? Just jumping in there and not giving a thought to the step after the next. Managing complexity? JAGNI. My tests passed, didn't they? And we can always refactor.

Getting it right takes discipline, research and experience. I prefer to get it right first time. Doesn't always work out as planned, but to me the need to refactor is a sign to stop and reassess what brought me to that point. And its always a result of having cut a corner somewhere.

Refactoring costs. And upfront design is about as fashionable as leg warmers.

I feel like we're talking about different things...

Continuous refactoring IS focusing on quality, it IS managing complexity. I think you've seriously misunderstood Deming if you think he stood for "Do it perfect from the start". The whole Kaizen philosophy is built on continuous improvement and elimination of waste (redundant code)

I'm not saying do a week/month/year of crappy coding and then "fix it later". Refactor continuously and as often as needed. Writing code is research.

The cost of refactoring is the greatest when you've done it too seldom. Otherwise with the right tools it's pretty efficient these days.

I see larger dangers in doing up front architect astronauting and make decisions before you've internalized all the problems.

For complex project it's of course appropriate to gather requirements, limitations etc and make a rough design, but going into too much detail is setting the cart before the horse... IMHO of course.

One of the coolest things I've learnt after over 20 years in this industry is to seek first to understand, and then to be understood.

We're talking about the same thing.

a3voices, you need to write a book on your technique for writing code correctly the first time. People have been trying to do that for as long as there have been programmers. You will make billions of dollars, guaranteed, assuming that you actually have a way to do that. Quit your job, drop whatever you're doing, and start writing that book right now, it will change the world. While you're at it, would you mind taking a look at the halting problem too? Thanks!

I guess no one understands sarcasm.