DonationCoder.com Software > MEWLO Web Framework
Mewlo Web Framework Blog
mouser:
I've been working hard on Mewlo for a few weeks now.. I think I'm going to start posting "blog"-like updates on this thread, discussing development progress and ideas -- if for no other reason than to keep a progress report. I'll try to post the first entry in the next day or so.
The starting point for more information on the Mewlo Web Framework is here:
* http://www.mewlo.com
mouser:
Mewlo Web Framework Blog Entry #1 - August 12, 2013
For my first entry in the Mewlo blog, it probably makes sense for me to discuss history and motivations.
I've been programming for 30+ years. My "native" programming language is C++, and I'm an unabashed proponent of object-oriented programming.
As much as I enjoy traditional application/desktop coding, more and more I find that the large project ideas I come up with involve creating custom "community" web sites and services -- that is, services where users sign up and create accounts and profiles online, create and share projects and interact with one another.
The primary roadblock to creating a novel, custom community website, is building the "boring" foundational structures to support the site. The user and group accounts, the infrastructure to deal with spammers and logging, moderators, performance issues, etc.. There is an absolutely astounding amount of foundational work required to build a non-trivial community website -- in fact this foundational code vastly exceeds the "custom" new code that one would write for most custom sites.
This foundational code is not just massive in size, but is full of difficult design decisions that effect efficiency, maintainability, and extensibility. It's code that is frequently implemented poorly in custom sites precisely because its the "boring precursor" work, not the heart of the new interesting ideas being implemented. It's code that gets reinvented over and over and over again on the web.
Now, there are a great many web "frameworks" and extensible content management systems in active development, and these frameworks are attempting to provide SOME of this common infrastructure.
Unfortunately, I have not found anything in existence which I have felt really good about using.
That is not to say that there aren't some truly excellent frameworks available. The closest to my interests is probably Django, but Pyramid (also Python) is another strong choice. Php also has some strong contenders, like Drupal (a content management system but also arguebly a web framework), CodeIgniter, and Yii (which I was so fond of that I used as the starting point for my first version of this project). I'm less familiar with the Java, C#, and more esoteric frameworks. And of course there is Ruby on Rails.
It's not that these frameworks aren't excellent -- it's that they have different philosophies and different goals. For the most part they are designed to be general purpose web frameworks that could be used to produce any kind of web service. And because of that, the user/group/moderator stuff does not get treated as a first-class concern. Such features are relegated to multiple competing plugins and addons, which are often out of date and don't work well with each other. Or they go too far in the other direction, and are primarily content management systems that are designed for non-coder administrators.
What I have in mind is a framework that is somewhere in between these all-purpose lower-level web frameworks on one end, and dedicated content management systems on the other.
Furthermore, my experience using frameworks and libraries has taught me that if you are coding something where the library can be used without modification, then it can be a huge time saver. But as soon as you need to make changes to the framework/library to bend it to get it to do things it wasn't strictly designed for, you pay an enormous cost in terms of code maintenance effort and code clarity.
There is a real benefit to starting from scratch and building a framework with a singular vision and focus and consistency. While the initial coding task is huge -- I believe there is a very real potential reward in terms of producing a framework that is better designed to solve this particular program -- providing the foundational infrastructure for custom user-driven community web services.
I also know from experience that even when there are some good existing tools, as a project matures it tends to find a useful niche and naturally differentiates itself from the alternatives. I am confident that over time, a new web framework like the one I am describing would find a place for itself separate and apart from the other web frameworks.
Creating such a large project from scratch, and being able to control exactly how things are structures and organized, is one of the most enjoyable things a programmer can be involved in. A good part of the motivation for working on this project is the pure enjoyment of that.
As a programming project -- this is a huge challenge and a giant project. This is the second time I've actually started working on it. The first time I started was about a year or so ago, under the name "Yumps", written in PHP.
For the last month or so I have restarted the project, this time in Python, under the name "Mewlo". I don't love Python -- but I think it's currently the best choice for a large web framework (one of the aims of this project is to create not just the code base for this framework, but the object structures and API so that it could be ported to a new better language in a few years).
I have created a GIT open source code repository for the Mewlo project and have begun uploading code there (see this thread for more info and links).
What I would really LOVE to do is find some more serious Python coders who are interested in being part of the team creating this. It's a huge project and it's going to need all the help it can get.
In the next blog entry I will talk a little about the organization and structure of the Mewlo system as I currently envision it.
mouser:
Mewlo Web Framework Blog Entry #2 - August 29, 2013 - A Digression
I was talking to another DonationCoder member recently about offering web services to people on the site, and it struck a cord with me that I'd like to share, that is relevant to Mewlo and coding in general.
If I have learned one lesson over and over again in my 7+ year experience with DonationCoder, it's to fear maintenance costs.
Now as a programmer, this should not be news to me -- the famous "Mythical Man Month" book talks about maintenance costs quite a bit, and how a substantial amount of time is spent maintaining code.
But different projects, attitudes, and lifestyles may put you in very different positions in terms of how much you have to worry about maintenance. Sometimes we are in positions, by choice or fate, where we do not feel the pain and fear of maintenance issues directly or often. Other times we find ourselves almost accidentally committed to supporting a system the requires near constant maintenance and care.
When we decided to upgrade the server that runs this website, we had the swell idea of pooling together money from people on the site, and then providing hosting services to our members. We hosted websites, email, etc. It's really not too much trouble to set up -- and we had a brilliant site administrator who was gung ho to set up any kind of custom setup people would want. What I didn't anticipate was how much work would be involved in the maintenance of such things, and how difficult it would be to walk back from such an idea. Whether it's worrying about operating system upgrades, dealing with outages, worrying about backups and failures, worrying about malicious users, etc. It gives me cold sweats just thinking about it.
Some people are intuitively suspicious of doing anything -- adding any feature, because they have a voice in their head that tells them that any added feature or service brings with it the potential for maintenance headaches.
Now by nature I'm the kind of person inclined to add features to software if I think it might be a useful addition. I tend not to worry too much about the increase in complexity of code to maintain, or the increased complexity of the configuration options. But I'm increasingly aware of the price to be paid for such decisions -- a price in terms of complexity of configuration dialogs, and a price to be paid for the growth of the code base.
I'm not suggesting a course of action here -- just advocating for a very healthy fear of maintenance costs -- and of keeping such things always in the forefront of your design decisions.
I can say that such fears are foremost in my thoughts when thinking about the DonationCoder site and in thinking about the Mewlo web framework. Over the last couple of years I've talked with people on DC about setting up a standalone "Self-Teaching Programming School" to replace the one that we have now on the site. And talked and experimented with moving the site over to a proper content management solution.
In such discussions I find myself in the regular position of having to throw water on a proposed idea that, to the person suggesting it, seems like "not too much work to set up". The problem of course is not that some new solution would be too much work to set up, but that the maintenance costs and risks associated with it are unbearable in the long term.
I'm particularly skeptical of solutions that involve connecting up multiple independent components made by different parties. My experience is that the more moving parts, the more frequent the incompatibility problems come up. And I've learned that even minor manual tasks (like merging in some changes into some source code) soon become painful and error prone when they have to be done over and over again, or after long delays when one has forgotten the steps needed.
And I'm especially skeptical about anything that requires specialty knowledge to do, because it means that the people involved in maintaining the system become indispensable.
That's another lesson I've learned -- avoid any solution where the steps required to maintain it are not codified into a document that can be passed on to a replacement maintenance person.
So let me wrap up by relating all of this to the Mewlo web framework. I think these lessons effect the framework in two very different ways..
The first way is in the design of Mewlo itself. Clean, documented, maintainable code has to be the primary organizational principle for the entire system. That's mostly common sense.
Slightly more controversial is my skepticism about the risks associated with interacting components maintained by different groups. That has led me to a fairly broad classification of certain "modules" in Mewlo as being part of the core system. Mewlo includes core "official" modules that many web frameworks delegate to unofficial, interacting (and often existing as multiple implementations written by different teams, and often abandoned). These are things like user groups/messaging/profiles/etc.
But the second way in which Mewlo is driven by this concept of a healthy fear of maintenance is in the role Mewlo is trying to play on the web. It would be fair to say that the primary mission of Mewlo is to ease the job of maintenance for creators of community-driven web services.
Supporting maintenance is exactly the thing that Mewlo is trying to do better than existing frameworks. The goal is to implement all of the support structures that are key to maintaining a site and keeping it running smoothly. This means extensive support for logging, upgrading, detecting and alerting on problems, monitoring operating conditions and detecting bad behavior or performance problems, and it means providing tools for moderators and administrators so that they can do their jobs easily.
You might say that Mewlo is designed for the Maintainers, and those who have a healthy fear of the cost of maintenance.
mouser:
Mewlo Web Framework Blog Entry #3 - October 2, 2013 - Where to Start?
I've been writing code for the Mewlo framework off and on for a couple of months now, but I think we still need one more preliminary blog post before I start getting into the details of the code.
What I want to talk about in this blog entry is establishing a solid foundation for a collaborative (open source) project.
Now I have to say off the bat that I don't have much experience with heading up active collaborative open-source projects. I have written some open source projects in the past, but these projects were never popular enough to get much collaboration, and not much thought went into creating an environment that fostered collaboration.
But for this project, it's very important to have an environment that both encourages collaboration and ensures that the the ecosystem around Mewlo does not become chaotic.
In the past I have been highly critical of some large open source projects, primarily those that support 3rd-party community addons or extensions. I find that such projects often become dumping grounds for large numbers of half-finished, abandoned, unsupported, conflicting addons, making it near impossible for anyone to find what they need and maintain a system built from these addons.
I am much more amenable to the idea of an actively "curated" ecosystem. That is, I am in favor of having some centralized organization and supervision of addons, and a fairly strict set of guidelines, policies, and standards.
An Open Source Repository
First, we choose an open source repository system. There are many to choose from -- I'm not sure that it matters terribly which one we go with, but for Mewlo I chose github. You can browse the Mewlo project on github here. Github has a number of attractive features, including a systematic way for people to propose code patches, and have them reviewed and then merged.
While we don't have need of it yet, we will eventually be making use of the issue tracker and wiki services provided by github.
Coding Style Document
The next thing we need to do is establish a coding style guide and set of standards for the code. The specific decisions about coding style are not critical -- but what is important is that all code conform to a single consistent style guide. Furthermore, we need to outline the philosophical approach to the way code is modularized and structured. A key aim of this project is to create a consistent, clean, unified framework that does not feel like it's the result of jamming together a bunch of separate pieces written by different people.
So it's important to have an easy-to-read document describing these guidelines. You can read the early draft of this document for Mewlo here. It still needs a lot of work and it's going to change over time, but I think the important thing is to have a place where we can refine these guidelines, and have an official document that people can refer to.
The website (http://www.mewlo.com) where that document, and all other Mewlo documentation, lives, is generated dynamically from documentation files and source code files. These files are part of the open source repository, and because of that they are always up to date with the actual source code. There is no such thing as the documentation reference manual falling out of sync with the source code.
Code the Infrastructure to Modularize the Project
Everything I've said in this post up till now is sort of textbook open source project management. But now I'm going to talk a little about something that doesn't come up all that often, about the idea of where to start coding.
Mewlo is a huge project, with many components. Where to start?
I've approached this question by asking myself what core infrastructure of code is needed for other coders to be able to jump in and write (and maintain) semi-independent modules?
In answering that question, I have focused on the following:
* 1. Organize the entire codebase around the system for supporting addons/extensions/plugins. In many projects, the plugin-system is the last thing to be designed -- something glued on top that would let plugins add features. I take the opposite approach here, and put the extension system at the foundation. Whether one is talking about the core code modules, or 3rd party plugins, everything is managed by a central system that manages packages of code, their dependencies, version and author information, etc.
* 2. Related to the idea of starting with an extension/addon system -- we have support structures to facilitate communication between components. This includes a signal broadcasting/receiving system, a component registry, a settings registry, etc.
* 3. Another foundational component that is important in facilitating the ability of other coders to create semi-independent modules is a substantial logging/debugging/error-handling system. There are two things we aim for here. First, and less importantly, we want to make it easy for coders to log and display information to help them track down issues. Second, and more importantly, we want to establish a standard for handling errors and logging information. We want a rich set of support functions and a coding standard for exactly how different kinds of errors should be handled, how they should be returned, when we should throw exceptions, etc.
* 4. Support for Unit Testing. Unit testing is a fancy (silly) term for the sensible idea of building automated tests that can test all aspects of the software and ensure that all code is working as expected. It's particularly useful in finding when changes to one piece of code cause unexpected failures in other pieces of code.
The common theme here is that in this first stage of coding, we are trying to establish the ground rules, the style of the code, the conceptual approach to the division of labor, the error-handling policies, etc., so that the code that gets built on top of all this will be consistent with the overall vision of the project.
Gothi[c]:
Good job!
In many projects, the plugin-system is the last thing to be designed -- something glued on top that would let plugins add features. I take the opposite approach here, and put the extension system at the foundation.
--- End quote ---
I think this is wise.
Navigation
[0] Message Index
[#] Next page
Go to full version