Author Topic: Planning a major programming project - tips requested (Read 12179 times)

Josh · « **on:** January 12, 2012, 09:21 PM »

All,

I am embarking on a new programming project. As such, I have decided it best to do a bit of fore-planning prior to the start of this. I am wondering what tips the community has for starting a new programming project.

Right now, my first thought is to plan out different modules that the program will require. Right now, my program will be divided into a series of core functions, each of which will perform different actions based on the context of the application the user finds themselves in. This tool will be a utility to help manage personnel data for a user, records, and various "databases" that most in my line of work find themselves encountering on a daily basis. Later, I might move into a multi-user model (not hard, just add in a "section manager" module). Right now, I can count around 10-12 different core functions to be included in this application. A side note: individual applications would not be beneficial given the intended nature of the project.

That said, I have several core questions I am looking to get answered.

The first is in regards to data file formatting and the best ideas for managing this. In the past, most of my smaller apps have used their own storage format, which I created using basic file i/o commands. Speed is NOT a requirement for this application as the files will not be insanely large nor will they require "instantaneous" access. I have been toying with learning something like sqlite, but am wondering if this will provide any REAL benefit given my usage scenario.

The next question is when to begin UI planning. To me, in the past, I have always designed a core GUI and developed based on the module I am looking at, at the time. I then proceed based on a list of functions that still need to be developed. When should this start? I do not want the UI to feel tacked on. I've seen too many applications where the UI feels like an after thought. Then again, I have also seen apps with beautiful interfaces but no access to core functions, or even very limited functionality. Thoughts?

Third, Are there any tools available for programmers to help visualize data models to be used by a program, manage modules requiring code and current status of development, etc? I would imagine such a tool exists, but I have never looked for one in the past.

That said, I am open to any ideas for help in planning a programming project. Any information, thoughts, or comments are appreciated and I feel this will help other coders by allowing them to refine their planning capabilities.

Thanks!
Josh

mouser · « **Reply #1 on:** January 12, 2012, 10:23 PM »

I have been toying with learning something like sqlite

It sounds like the first real question you have to answer is -- are you talking about saving application settings/options? or a large database of data that can be thought of as many rows of a table with some fixed columns/fields. If not the latter, than you don't want sqlite or any database engine.

Consider something like xml for options -- using a library to help you load and save xml files.

Are there any tools available for programmers to help visualize data models to be used by a program

There is a whole industry devoted to things like this, UML tools (Universal Modeling Language) -- but there are some real debates about the usefulness of such elaborate mechanisms when working on small projects or single/two man teams.

Personally i don't find these formal tools very useful. It's hard to beat the value of pen and paper.

Half of programming is PLANNING. I think it's a very common mistake of new coders to think they are supposed to just sit down and write code off the top of their head.

Josh · « **Reply #2 on:** January 12, 2012, 10:28 PM »

I have been toying with learning something like sqlite

It sounds like the first real question you have to answer is -- are you talking about saving application settings/options? or a large database of data that can be thought of as many rows of a table with some fixed columns/fields. If not the latter, than you don't want sqlite or any database engine.

Consider something like xml for options -- using a library to help you load and save xml files.
-mouser (January 12, 2012, 10:23 PM)

[/quote]

This will be a large file containing user data. There will be external files stored in their raw formats (jpeg, pdf, etc). I was debating about an already designed database system like sqlite, but also know that I have written my own file save procedures in the past and it has worked out nicely.

I was advised AGAINST XML because of the sheer complexity of implementation. There is no desire to export this data, as my application will be the first in its industry.

Are there any tools available for programmers to help visualize data models to be used by a program

There is a whole industry devoted to things like this, UML tools (Universal Modeling Language) -- but there are some real debates about the usefulness of such elaborate mechanisms when working on small projects or single/two man teams.

Personally i don't find these formal tools very useful. It's hard to beat the value of pen and paper.

Half of programming is PLANNING. I think it's a very common mistake of new coders to think they are supposed to just sit down and write code off the top of their head.

This is why I want to start off right. I want to plan this properly, but I have to answer those questions above. I am looking for tips from the pros, doco users, and am hoping to do this project right.

Josh · « **Reply #3 on:** January 12, 2012, 10:46 PM »

Also, I would like to add what I somehow forgot to ask....What steps do YOU, as coders, take when you are planning a project?

Ath · « **Reply #4 on:** January 13, 2012, 01:55 AM »

I was advised AGAINST XML because of the sheer complexity of implementation. There is no desire to export this data, as my application will be the first in its industry.
-Josh (January 12, 2012, 10:28 PM)

Have you settled on a programming language/environment already? That heavily influences a lot of choices.
If working with Visual Studio, XML is a very natural choice for storing application settings, as xml is much easier to use then either .properties or .ini files there, but if working with a LAMP or WAMP environment or Java (both pojo or an app container), .properties or .config would be easier. Working with some Borland (yeah I still hang onto the original name) stuff like C++(Builder) or Delphi, .ini files are a quite logical choice.
(Please share your choice/thoughts)

Storing data, especially if it's most 'other type of files', is imho best saved (copied) as the original filetypes, with a light database or xml used as a glue pointing to the files stored in a directory structure, off of a single 'mount point'.
If the data where most user-entered, with an occasional file attached, I'd go for a proper SQL database, like MS-SQL Express edition (2008R2 Express allows for 10 GB storage per database), PostgreSQL or MySQL (the disky is the limit

).

What steps do YOU, as coders, take when you are planning a project?
-Josh (January 12, 2012, 10:46 PM)

I usually start with a few PoC's to prove any technical challenges will actually work as intended, then top-down, dividing the project into smaller chunks (until split down to single lines of sourcecode

)
[quoth=Albert Einstein]"Everything should be made as simple as possible, but not simpler."[/quoth]

tranglos · « **Reply #5 on:** January 13, 2012, 09:23 AM »

Also, I would like to add what I somehow forgot to ask....What steps do YOU, as coders, take when you are planning a project?
-Josh (January 12, 2012, 10:46 PM)

Step one: stop planning, go ahead and start coding. Seriously. The moment you write the first line of code you will start getting a much better understanding of (a) what it is you are making, and (b) what difficulties to expect. And the more code you write, the better your understanding will be.

Step two: plan to throw away the first attempt. All of it. Plan to change the language, to change the data storage mechanism, change everything. This goes together with step 1, because the sooner you begin, the easier it will be to let go of what you've done and start anew with a better grasp of what the project entails. I'm not saying you will throw it away; I'm saying it happens (or should happen) most of the time.

Or, if you're not quite ready to start just yet and don't like the idea of throwing away several months' worth of work, then start coding little "testbed" projects right away. Little parts of your future app, its individual subsystems. You need to know what IO mechanism fits your app best, so just write the IO code using XML or SQLite (for example) and see how it feels. Then throw away the testbed code, because it will probably be quick-and-dirty, but by then you will know what works for you, for this specific task.

That comes from my experience, YMMV, but I have seen similar advice given by truly seasoned programmers. I imagine there are pros out there who can design everything in their heads and then just do the mechanical work of putting it in code, but it certainly doesn't work that way for me. It's exceedingly hard to see all the possible pitfalls, problems, design decisions you'll have to make until you write the actual code - and then it becomes obvious. (Maybe it's just me, I suck at chess and that's kind of similar.)

I have thrown away much more code than is running in the apps I've ever published. I was designing KeyNote 2.0 "mentally" for four years, I literally dreamt about it, before I decided I wasn't up to it (and other things intervened). But most of all it was some sort of fatigue coming from several years of living with the abstract ideal of what the new version was supposed to be, without really doing anything about it.

I have another vaporware project that I've been designing since 2001. That's right, almost 11 years now

Someone called this "analysis-paralysis" and that's quite apt. All the projects that amounted to anything have been projects that I just sat down and wrote like there was no tomorrow.

mouser · « **Reply #6 on:** January 13, 2012, 09:34 AM »

I have found that the "plan to throw one away; you will, anyhow." philosophy is extremely insightful but often impractical.

I think the key thing to accept is that you really will learn more in the first steps of coding an application than you can anticipate with planning ahead of time. And often the very very best way to plan and refine ideas is to start writing a throwaway first test version. And throwing away the first version is probably the very very best way to produce better cleaner code.

But often that's just not a practical option -- and furthermore, I do think there is some real value in planning out an application -- just sitting down with paper and pencil and imagining all of the operations and the workflow. If you simply sit down and code until your application is done and expect that first version to be well organized you will end up in trouble.

Perhaps another way of saying this is that planning is critical -- but you can "plan" by jumping in and writing a first version to throw away, or you can plan before you ever start coding.

However the comments about "analysis paralysis" or planning paralysis, are on point too -- I have frequently gotten held up for astonishingly long delays as I ponder the "right" way to do things. Much of the time it's just wasted delay and I would have been better off just coding a paralysis. Unfortunately, the other half of the time the delay has resulted in a rethinking of the problem which I would not have been able to do if I just jumped in and started coding.

The truth of the matter is, that much of what differentiates an experienced coder vs. someone just starting out is not knowledge of a language or implementation speed -- it's developing good judgement and skills regarding how to attack a problem. And you can only learn that through experience and practice.

tranglos · « **Reply #7 on:** January 13, 2012, 09:46 AM »

This will be a large file containing user data. There will be external files stored in their raw formats (jpeg, pdf, etc). I was debating about an already designed database system like sqlite, but also know that I have written my own file save procedures in the past and it has worked out nicely.

I was advised AGAINST XML because of the sheer complexity of implementation. There is no desire to export this data, as my application will be the first in its industry.
-Josh (January 12, 2012, 10:28 PM)

I don't think you can underestimate interoperability and ease of access. Users will thank you for it, if they ever need to access the data outside of your app.

I wouldn't say XML is complex, as long as you can use a 3rd-party parser that does the heavy lifting for you. But it is (a) relatively slow to read (b) wasteful when the tags take up more bytes than user data (c) potentially memory intensive, depending on the parser and how you use it; (d) brittle, because a single wrong byte may prevent your app from loading any of the data (that's particularly important if your users may ever fiddle with the data files themselves).

But XML is very useful and flexible when you need or expect to (a) represent hierarchical data; (b) store data of different types, including binary; (c) modify and expand the capabilities of your app or your data files. You can easily add a new tag to store a new piece of data, which earlier versions of your app may just ignore without breaking. Doing the same in custom binary files takes more work, as you have to introduce some sort of "file version" marker and check for it all the time (version 1.1, so expect a date now; version 1.2, time is now stored as UTC; version 1.3, added a comment field here...) That last thing is much easier to do with XML.

You could also design your custom "binary" format that is structured like XML, with numeric identifiers instead of tags. That worked for me once, but after that I decided SQLite was easier and faster and more capable to begin with.

SQLite is really neat in that it doesn't need a server or any complex installation (or any db license). If you use MySQL, who's going to install it, and who's going to do tech support for MySQL issues, for example? For me, the main advantage of a database is that you can read and write small portions of your data instead of the whole datafile at once. And you get a lot of built in logic for free via SELECT queries that you would otherwise have to devise yourself. Frequent disk access is an obvious downside, although you can configure your SQLite database to stay in RAM (by specifying a sufficient cache size or even explicitly copying the db to RAM).

Another downside of SQLite for me is: interminable query strings everywhere. I do keep them all in once place actually, but unless you generate all of your SQL at runtime, you may end up with hundreds of SELECT statements that are hard to maintain. If you add a column to a table, you need to go through all of them to see which ones must be modified. I hate that part, and you can of course parametrize the queries and glue them together from pieces, but the pieces still remain as string constants, and that's bad. Still, right now I wish I'd written almost all my apps with an SQLite backend, and I'll be using it for practically every future project I can think of. It's really flexible, and by far the fastest database engine on the desktop.

tranglos · « **Reply #8 on:** January 13, 2012, 10:10 AM »

I have found that the "plan to throw one away; you will, anyhow." philosophy is extremely insightful but often impractical.
-mouser (January 13, 2012, 09:34 AM)

I agree. And there always comes a point beyond which you will not be willing to throw all the code away, because rewriting it from scratch would just take too long. That's why I'm suggesting coding the little self-contained parts first. To try out what works and how it works. You can skip much error checking or knock yourself out with exceptions if you prefer, you don't care about the UI, you can pepper the code with verbose logging and profiling, all the things that will help you learn what you are doing, without a care in the world. I suppose it is a personal thing, but it's very specifically the method that has allowed me to go back to creating working apps after a five or so years of feeling totally not-up-to-it.

But often that's just not a practical option -- and furthermore, I do think there is some real value in planning out an application -- just sitting down with paper and pencil and imagining all of the operations and the workflow. If you simply sit down and code until your application is done and expect that first version to be well organized you will end up in trouble.

That last thing is very true. But what I'm trying to say is that planning is extremely hard - for me it is, anyway. A pencil and a napkin, sure, all the time. But in my experience I don't even know very well what to name my classes and methods before I actually type it all up see what it means and how it relates to the existing code (without necessarily compiling and running it). Probably half the time I spend coding is refactoring the things that initially came out of my head. Refactoring is typically understood as a technique for maintaining code, but for me it's a huge part of creating it in the first place. I have to see the code to really understand it. Once as a kid I got a serious knock on the back of my head, maybe that's why

Perhaps another way of saying this is that planning is critical -- but you can "plan" by jumping in and writing a first version to throw away, or you can plan before you ever start coding.

And at least be prepared to throw away your plans

40hz · « **Reply #9 on:** January 13, 2012, 10:58 AM »

The truth of the matter is, that much of what differentiates an experienced coder vs. someone just starting out is not knowledge of a language or implementation speed -- it's developing good judgement and skills regarding how to attack a problem. And you can only learn that through experience and practice.
-mouser (January 13, 2012, 09:34 AM)

So from that I guess you're saying the real difference between an experienced and a beginner coder is...experience?

Good to know!

Kidding...just kidding.

skwire · « **Reply #10 on:** January 13, 2012, 11:18 AM »

I just start randomly hitting the keyboard and hope it compiles at the end.

app103 · « **Reply #11 on:** January 13, 2012, 11:44 AM »

Since most of my applications do have a GUI, I begin there, designing it in my head, imagining myself as the user and using the application mentally. When I feel comfortable with the mental image, I begin drawing it on paper, with notes as to what each control will do when activated and other text based notes about its unseen inner workings.

Then before I ever open my IDE, I give a full explanation of my idea to someone else that I consider an experienced coder. I highly recommend mouser's Idea Explosionifier, even if you want to keep things simple. It will help you to avoid coding yourself into a corner if you ever decide to add any of the plethora of additional features he will think up. Be careful though, it's easy to get caught up in the thought that you have to turn your project into the uncodable nightmare that will be produced by the Idea Explosionifier. You don't. You shouldn't. If you even consider trying, you'll just become overwhelmed and get discouraged and not even bother with the project. Just consider the future possibilities and make sure you leave room for adding them.

Then work on the GUI of your app only. Make a screenshot and run it through the Idea Explosionifier again. The results may be very different than before.

If this is going to be a large project, it's likely that the Idea Explosionifier will come up with about 10 years worth of ideas to keep your project alive.

wraith808 · « **Reply #12 on:** January 13, 2012, 01:51 PM »

Since most of my applications do have a GUI, I begin there, designing it in my head, imagining myself as the user and using the application mentally. When I feel comfortable with the mental image, I begin drawing it on paper, with notes as to what each control will do when activated and other text based notes about its unseen inner workings.
-app103 (January 13, 2012, 11:44 AM)

From a different perspective, I design my UI last, and my data first. Sometimes I find that GUI implementations can limit your functionality and data model by the fact that they are already constrained by preconceptions put in place by the UI. Then once I have my data modeled, I base my UI design around the primary entrance point into the data, i.e. if it is something contact based, the primary entry point might be a person's basic data.

Taking that contact paradigm a little further to illustrate what I mean, I might design the contact form to have the address information built in if I start UI first. If I start data first, then I might see that I have a need to have many addresses per user, so my UI design might then follow with the address information on some sort of container that can be changed out depending on which address is being utilized.

Just my .02.

Ath · « **Reply #13 on:** January 13, 2012, 02:07 PM »

@Josh, any reply or reaction or are you just overwhelmed by all these nice responses?

TaoPhoenix · « **Reply #14 on:** January 13, 2012, 02:09 PM »

Okay, while it's not "programming", I have something worth at least 1.7 cents if not two cents.

For me, a bit of "dive in" needs equal parts of planning. I am redesigning an estimating database so that the new structure matches new accounting codes, but our lead estimator still wants it to look and feel like it does now so he can keep his throughput up.

It depends on your intuition - right away I noticed variants on Many-to-One and One-to-Many logic puzzles. But rather like files in windows, you can't just make them "all the same name" so it calls for an intermediate shuffle step.

For me the big danger of big projects is that you can "fake it" some 75% of the way there, then a design decision just sinks you. If you spent a ton of time doing "similar steps" for all the cases, yeah, I'm no fan of throwing it all out.

Instead, what "feels like the scary part"? Maybe you know how 75% of the program is a snap, just requiring ... beverages... for motivation, but is some piece of core logic really brittle? The more interlocking stuff is, the more it risks creaking at the seams when you want to do Version 2.0 when a new feature becomes a Must-Have.

Josh · « **Reply #15 on:** January 13, 2012, 02:29 PM »

I am taking all of this in and trying to determine how I want to proceed. My biggest hurdle is all of the PLANNED functionality i want in the app. Right now, I am developing a todo list of what functions I want the app to perform, and what data is associated with each data format. As I look at it, a database might be the option for actual data storage, an INI for settings and I will expand as needed. I do not want to do XML because, as I said above, this is the only app of its type and there will be no need to export it. The information entered will be from already existing documents that most personnel in my field already keep in paper form (ALWAYS CYA, as they say).

I think that once I determine functionality, what will exist in the initial release, and how the data will be formatted, I am going to proceed. GUI is still the big thing I want to discuss as I have no idea when is the appropriate time to begin design.

tranglos · « **Reply #16 on:** January 13, 2012, 02:55 PM »

I highly recommend mouser's Idea Explosionifier
-app103 (January 13, 2012, 11:44 AM)

LOL, this is epic. And true!

TaoPhoenix · « **Reply #17 on:** January 13, 2012, 05:40 PM »

...this is the only app of its type and there will be no need to export it. The information entered will be from already existing documents that most personnel in my field already keep in paper form (ALWAYS CYA, as they say).

I think that once I determine functionality, what will exist in the initial release, and how the data will be formatted, I am going to proceed. GUI is still the big thing I want to discuss as I have no idea when is the appropriate time to begin design.
-Josh (January 13, 2012, 02:29 PM)

Couple of thoughts.

I do recommend you plan in an export path. I and many others have concerns about Locked Gardens, wherein everything is all hunky dory until life makes you move on, then no one after you can extricate the data from the proprietary saving format. At my company I recommended twice against these kinds of "data black holes" in favor of "closer to the metal" just keeping files and then adding intelligence on top of that.

As for GUI, I'm far from an expert, so get your salt ready, but what about just start with a bunch of menus as placeholders? For GUI's you're going to get into the whole Ribbon discussion, so have a meta-answer For-or-Against the Ribbon. (Vote here: Against Ribbon, mostly.)

Josh · « **Reply #18 on:** January 13, 2012, 08:38 PM »

The issue with data format and export is that, again, this is related to personnel records. All of the entered data exists in paper form, or digital copy depending on where it is obtained from. No tool that I've ever seen utilizes this data other than for individual download. There are no importing tools, and as such any plans for exporting will be at the very bottom of the list in terms of functionality. Not saying I won't ever implement it, but I am saying that I have no plans for the discernable future. While I will always keep it in mind, I have no plans to add it due to the nature of the information being handled. This is not like movie databases or mp3 collection managers, there information here is personal in nature, regarding to progression within my job field, and no tools exist which would make use of the types of data I plan on storing in the app. All of the individual files (training certs, etc) will be stored as their individual raw formats (jpg, pdf, tiff, etc) and as such there will be no lock-in. All of the files will be completely transportable. With this in mind, export functionality is the very last priority on my list.

ecaradec · « **Reply #19 on:** January 14, 2012, 06:49 AM »

I went from the plan everything route then back to no planning at all. When I'm working on personnal projects, I'm very iterative now, to the point that I usually start with everything in one CPP file, then gradually move things into headers, then split headers in H and CPP. I try to reuse functionnality available in the OS and the libraries as much as possible if they don't make things more complex. I often have a pretty clear idea of what design I expect to work, so I often test that.

I take care to separe the pieces that effectivelly do something and the part that are structure. A wrong structure can be extremely damaging to a project and from what I've experienced I'm much more sensitive to go into something overly complex by planning how your project is design before getting my hands dirty. Structure is best when there is as few as possible of it. Keeping the pieces that do the work apart make it much easier to change the structure too, so this is important (don't glue everything inside deep structure hierarchy ).

All of that is learned from experience though, I don't know how much of this is transferable to anyone. I tried to explain my may to a coworker and he went into a template berzerker rage

. I now have years of rework to get things back in order.

My current rule is now : "Don't listen too much the tips of people". If something doesn't make sense for you don't try to use it, experiment. XML can be good and bad, and INI files, sqlite or not. All of theses choices depends on very small scale decisions. If at some point a decisions make things harder, it's most likely wrong for you.

capitalH · « **Reply #20 on:** January 16, 2012, 12:44 AM »

This will be a large file containing user data. There will be external files stored in their raw formats (jpeg, pdf, etc). I was debating about an already designed database system like sqlite, but also know that I have written my own file save procedures in the past and it has worked out nicely.

I was advised AGAINST XML because of the sheer complexity of implementation. There is no desire to export this data, as my application will be the first in its industry.
-Josh (January 12, 2012, 10:28 PM)

I don't think you can underestimate interoperability and ease of access. Users will thank you for it, if they ever need to access the data outside of your app.

I wouldn't say XML is complex, as long as you can use a 3rd-party parser that does the heavy lifting for you. But it is (a) relatively slow to read (b) wasteful when the tags take up more bytes than user data (c) potentially memory intensive, depending on the parser and how you use it; (d) brittle, because a single wrong byte may prevent your app from loading any of the data (that's particularly important if your users may ever fiddle with the data files themselves).

But XML is very useful and flexible when you need or expect to (a) represent hierarchical data; (b) store data of different types, including binary; (c) modify and expand the capabilities of your app or your data files. You can easily add a new tag to store a new piece of data, which earlier versions of your app may just ignore without breaking. Doing the same in custom binary files takes more work, as you have to introduce some sort of "file version" marker and check for it all the time (version 1.1, so expect a date now; version 1.2, time is now stored as UTC; version 1.3, added a comment field here...) That last thing is much easier to do with XML.

You could also design your custom "binary" format that is structured like XML, with numeric identifiers instead of tags. That worked for me once, but after that I decided SQLite was easier and faster and more capable to begin with.

SQLite is really neat in that it doesn't need a server or any complex installation (or any db license). If you use MySQL, who's going to install it, and who's going to do tech support for MySQL issues, for example? For me, the main advantage of a database is that you can read and write small portions of your data instead of the whole datafile at once. And you get a lot of built in logic for free via SELECT queries that you would otherwise have to devise yourself. Frequent disk access is an obvious downside, although you can configure your SQLite database to stay in RAM (by specifying a sufficient cache size or even explicitly copying the db to RAM).

Another downside of SQLite for me is: interminable query strings everywhere. I do keep them all in once place actually, but unless you generate all of your SQL at runtime, you may end up with hundreds of SELECT statements that are hard to maintain. If you add a column to a table, you need to go through all of them to see which ones must be modified. I hate that part, and you can of course parametrize the queries and glue them together from pieces, but the pieces still remain as string constants, and that's bad. Still, right now I wish I'd written almost all my apps with an SQLite backend, and I'll be using it for practically every future project I can think of. It's really flexible, and by far the fastest database engine on the desktop.
-tranglos (January 13, 2012, 09:46 AM)

I am more a data guy than a programming guy (statistician by trade) so here goes my 2c:

XML - very useful for hierarchical data, and easier to use than it looks at first. Though my first attempts failed miserably (when I attempted to do my own parsing) with MSXML (or potentially another library) it is dead easy. Speed is not an issue with smallish data (I used it successfully for 10 000 records, about 50 attributes per record, very hierarchical) - though I only use it during load/save - not updating it continuously.

sql (or other database) - very useful if you mostly work on subsections of the data, or if you have huge data and can with with a subsection at a time.

CSV - my personal favorite. Works everywhere, you decide the format. Open and transparent. For some protection - compress it before saving. Does not work that well with huge data (limited by memory usually). Does not work well with hierarchical data or data that can have a variable number of attributes per record (say for example you have a bunch of projects (records) and for a project a list of people working on it - you will probably have to have a cell that has a separator which)