Mewlo Web Framework Blog

[mouser]

I've been working hard on Mewlo for a few weeks now.. I think I'm going to start posting "blog"-like updates on this thread, discussing development progress and ideas -- if for no other reason than to keep a progress report.  I'll try to post the first entry in the next day or so.

The starting point for more information on the Mewlo Web Framework is here:

• http://www.mewlo.com

[mouser]

Mewlo Web Framework Blog Entry #1 - August 12, 2013

For my first entry in the Mewlo blog, it probably makes sense for me to discuss history and motivations.

I've been programming for 30+ years.  My "native" programming language is C++, and I'm an unabashed proponent of object-oriented programming.

As much as I enjoy traditional application/desktop coding, more and more I find that the large project ideas I come up with involve creating custom "community" web sites and services -- that is, services where users sign up and create accounts and profiles online, create and share projects and interact with one another.

The primary roadblock to creating a novel, custom community website, is building the "boring" foundational structures to support the site.  The user and group accounts, the infrastructure to deal with spammers and logging, moderators, performance issues, etc..  There is an absolutely astounding amount of foundational work required to build a non-trivial community website -- in fact this foundational code vastly exceeds the "custom" new code that one would write for most custom sites.

This foundational code is not just massive in size, but is full of difficult design decisions that effect efficiency, maintainability, and extensibility.  It's code that is frequently implemented poorly in custom sites precisely because its the "boring precursor" work, not the heart of the new interesting ideas being implemented.  It's code that gets reinvented over and over and over again on the web.


Now, there are a great many web "frameworks" and extensible content management systems in active development, and these frameworks are attempting to provide SOME of this common infrastructure.

Unfortunately, I have not found anything in existence which I have felt really good about using.

That is not to say that there aren't some truly excellent frameworks available.  The closest to my interests is probably Django, but Pyramid (also Python) is another strong choice.  Php also has some strong contenders, like Drupal (a content management system but also arguebly a web framework), CodeIgniter, and Yii (which I was so fond of that I used as the starting point for my first version of this project).  I'm less familiar with the Java, C#, and more esoteric frameworks.  And of course there is Ruby on Rails.

It's not that these frameworks aren't excellent -- it's that they have different philosophies and different goals.  For the most part they are designed to be general purpose web frameworks that could be used to produce any kind of web service.  And because of that, the user/group/moderator stuff does not get treated as a first-class concern.  Such features are relegated to multiple competing plugins and addons, which are often out of date and don't work well with each other.  Or they go too far in the other direction, and are primarily content management systems that are designed for non-coder administrators.

What I have in mind is a framework that is somewhere in between these all-purpose lower-level web frameworks on one end, and dedicated content management systems on the other.


Furthermore, my experience using frameworks and libraries has taught me that if you are coding something where the library can be used without modification, then it can be a huge time saver.  But as soon as you need to make changes to the framework/library to bend it to get it to do things it wasn't strictly designed for, you pay an enormous cost in terms of code maintenance effort and code clarity.

There is a real benefit to starting from scratch and building a framework with a singular vision and focus and consistency.  While the initial coding task is huge -- I believe there is a very real potential reward in terms of producing a framework that is better designed to solve this particular program -- providing the foundational infrastructure for custom user-driven community web services.

I also know from experience that even when there are some good existing tools, as a project matures it tends to find a useful niche and naturally differentiates itself from the alternatives.  I am confident that over time, a new web framework like the one I am describing would find a place for itself separate and apart from the other web frameworks.

Creating such a large project from scratch, and being able to control exactly how things are structures and organized, is one of the most enjoyable things a programmer can be involved in.  A good part of the motivation for working on this project is the pure enjoyment of that.

As a programming project -- this is a huge challenge and a giant project.  This is the second time I've actually started working on it.  The first time I started was about a year or so ago, under the name "Yumps", written in PHP.

For the last month or so I have restarted the project, this time in Python, under the name "Mewlo".  I don't love Python -- but I think it's currently the best choice for a large web framework (one of the aims of this project is to create not just the code base for this framework, but the object structures and API so that it could be ported to a new better language in a few years).


I have created a GIT open source code repository for the Mewlo project and have begun uploading code there (see this thread for more info and links).

What I would really LOVE to do is find some more serious Python coders who are interested in being part of the team creating this.  It's a huge project and it's going to need all the help it can get.

In the next blog entry I will talk a little about the organization and structure of the Mewlo system as I currently envision it.

[mouser]

Mewlo Web Framework Blog Entry #2 - August 29, 2013 - A Digression

I was talking to another DonationCoder member recently about offering web services to people on the site, and it struck a cord with me that I'd like to share, that is relevant to Mewlo and coding in general.

If I have learned one lesson over and over again in my 7+ year experience with DonationCoder, it's to fear maintenance costs.

Now as a programmer, this should not be news to me -- the famous "Mythical Man Month" book talks about maintenance costs quite a bit, and how a substantial amount of time is spent maintaining code.

But different projects, attitudes, and lifestyles may put you in very different positions in terms of how much you have to worry about maintenance.  Sometimes we are in positions, by choice or fate, where we do not feel the pain and fear of maintenance issues directly or often.  Other times we find ourselves almost accidentally committed to supporting a system the requires near constant maintenance and care.

When we decided to upgrade the server that runs this website, we had the swell idea of pooling together money from people on the site, and then providing hosting services to our members.  We hosted websites, email, etc.  It's really not too much trouble to set up -- and we had a brilliant site administrator who was gung ho to set up any kind of custom setup people would want.  What I didn't anticipate was how much work would be involved in the maintenance of such things, and how difficult it would be to walk back from such an idea.  Whether it's worrying about operating system upgrades, dealing with outages, worrying about backups and failures, worrying about malicious users, etc.  It gives me cold sweats just thinking about it.

Some people are intuitively suspicious of doing anything -- adding any feature, because they have a voice in their head that tells them that any added feature or service brings with it the potential for maintenance headaches.

Now by nature I'm the kind of person inclined to add features to software if I think it might be a useful addition.  I tend not to worry too much about the increase in complexity of code to maintain, or the increased complexity of the configuration options.  But I'm increasingly aware of the price to be paid for such decisions -- a price in terms of complexity of configuration dialogs, and a price to be paid for the growth of the code base.

I'm not suggesting a course of action here -- just advocating for a very healthy fear of maintenance costs -- and of keeping such things always in the forefront of your design decisions.


I can say that such fears are foremost in my thoughts when thinking about the DonationCoder site and in thinking about the Mewlo web framework.  Over the last couple of years I've talked with people on DC about setting up a standalone "Self-Teaching Programming School" to replace the one that we have now on the site.  And talked and experimented with moving the site over to a proper content management solution.

In such discussions I find myself in the regular position of having to throw water on a proposed idea that, to the person suggesting it, seems like "not too much work to set up".  The problem of course is not that some new solution would be too much work to set up, but that the maintenance costs and risks associated with it are unbearable in the long term.

I'm particularly skeptical of solutions that involve connecting up multiple independent components made by different parties.  My experience is that the more moving parts, the more frequent the incompatibility problems come up.  And I've learned that even minor manual tasks (like merging in some changes into some source code) soon become painful and error prone when they have to be done over and over again, or after long delays when one has forgotten the steps needed.

And I'm especially skeptical about anything that requires specialty knowledge to do, because it means that the people involved in maintaining the system become indispensable.

That's another lesson I've learned -- avoid any solution where the steps required to maintain it are not codified into a document that can be passed on to a replacement maintenance person.


So let me wrap up by relating all of this to the Mewlo web framework. I think these lessons effect the framework in two very different ways..

The first way is in the design of Mewlo itself.  Clean, documented, maintainable code has to be the primary organizational principle for the entire system.  That's mostly common sense.

Slightly more controversial is my skepticism about the risks associated with interacting components maintained by different groups.  That has led me to a fairly broad classification of certain "modules" in Mewlo as being part of the core system.  Mewlo includes core "official" modules that many web frameworks delegate to unofficial, interacting (and often existing as multiple implementations written by different teams, and often abandoned).  These are things like user groups/messaging/profiles/etc.

But the second way in which Mewlo is driven by this concept of a healthy fear of maintenance is in the role Mewlo is trying to play on the web.  It would be fair to say that the primary mission of Mewlo is to ease the job of maintenance for creators of community-driven web services.

Supporting maintenance is exactly the thing that Mewlo is trying to do better than existing frameworks.  The goal is to implement all of the support structures that are key to maintaining a site and keeping it running smoothly.  This means extensive support for logging, upgrading, detecting and alerting on problems, monitoring operating conditions and detecting bad behavior or performance problems, and it means providing tools for moderators and administrators so that they can do their jobs easily.

You might say that Mewlo is designed for the Maintainers, and those who have a healthy fear of the cost of maintenance.

[mouser]

Mewlo Web Framework Blog Entry #3 - October 2, 2013 - Where to Start?

I've been writing code for the Mewlo framework off and on for a couple of months now, but I think we still need one more preliminary blog post before I start getting into the details of the code.

What I want to talk about in this blog entry is establishing a solid foundation for a collaborative (open source) project.

Now I have to say off the bat that I don't have much experience with heading up active collaborative open-source projects.  I have written some open source projects in the past, but these projects were never popular enough to get much collaboration, and not much thought went into creating an environment that fostered collaboration.

But for this project, it's very important to have an environment that both encourages collaboration and ensures that the the ecosystem around Mewlo does not become chaotic.

In the past I have been highly critical of some large open source projects, primarily those that support 3rd-party community addons or extensions.  I find that such projects often become dumping grounds for large numbers of half-finished, abandoned, unsupported, conflicting addons, making it near impossible for anyone to find what they need and maintain a system built from these addons.

I am much more amenable to the idea of an actively "curated" ecosystem.  That is, I am in favor of having some centralized organization and supervision of addons, and a fairly strict set of guidelines, policies, and standards.

An Open Source Repository

First, we choose an open source repository system.  There are many to choose from -- I'm not sure that it matters terribly which one we go with, but for Mewlo I chose github.  You can browse the Mewlo project on github here.  Github has a number of attractive features, including a systematic way for people to propose code patches, and have them reviewed and then merged.

While we don't have need of it yet, we will eventually be making use of the issue tracker and wiki services provided by github.

Coding Style Document

The next thing we need to do is establish a coding style guide and set of standards for the code.  The specific decisions about coding style are not critical -- but what is important is that all code conform to a single consistent style guide.  Furthermore, we need to outline the philosophical approach to the way code is modularized and structured.  A key aim of this project is to create a consistent, clean, unified framework that does not feel like it's the result of jamming together a bunch of separate pieces written by different people.

So it's important to have an easy-to-read document describing these guidelines.  You can read the early draft of this document for Mewlo here.  It still needs a lot of work and it's going to change over time, but I think the important thing is to have a place where we can refine these guidelines, and have an official document that people can refer to.

The website (http://www.mewlo.com) where that document, and all other Mewlo documentation, lives, is generated dynamically from documentation files and source code files.  These files are part of the open source repository, and because of that they are always up to date with the actual source code.  There is no such thing as the documentation reference manual falling out of sync with the source code.

Code the Infrastructure to Modularize the Project

Everything I've said in this post up till now is sort of textbook open source project management.  But now I'm going to talk a little about something that doesn't come up all that often, about the idea of where to start coding.

Mewlo is a huge project, with many components.  Where to start?

I've approached this question by asking myself what core infrastructure of code is needed for other coders to be able to jump in and write (and maintain) semi-independent modules?

In answering that question, I have focused on the following:

• 1. Organize the entire codebase around the system for supporting addons/extensions/plugins.  In many projects, the plugin-system is the last thing to be designed -- something glued on top that would let plugins add features.  I take the opposite approach here, and put the extension system at the foundation.  Whether one is talking about the core code modules, or 3rd party plugins, everything is managed by a central system that manages packages of code, their dependencies, version and author information, etc.
• 2. Related to the idea of starting with an extension/addon system -- we have support structures to facilitate communication between components.  This includes a signal broadcasting/receiving system, a component registry, a settings registry, etc.
• 3. Another foundational component that is important in facilitating the ability of other coders to create semi-independent modules is a substantial logging/debugging/error-handling system.  There are two things we aim for here.  First, and less importantly, we want to make it easy for coders to log and display information to help them track down issues.  Second, and more importantly, we want to establish a standard for handling errors and logging information.  We want a rich set of support functions and a coding standard for exactly how different kinds of errors should be handled, how they should be returned, when we should throw exceptions, etc.
• 4. Support for Unit Testing.  Unit testing is a fancy (silly) term for the sensible idea of building automated tests that can test all aspects of the software and ensure that all code is working as expected.  It's particularly useful in finding when changes to one piece of code cause unexpected failures in other pieces of code.

The common theme here is that in this first stage of coding, we are trying to establish the ground rules, the style of the code, the conceptual approach to the division of labor, the error-handling policies, etc., so that the code that gets built on top of all this will be consistent with the overall vision of the project.

[Gothi[c]]

Good job!

In many projects, the plugin-system is the last thing to be designed -- something glued on top that would let plugins add features.  I take the opposite approach here, and put the extension system at the foundation.

I think this is wise.

[mouser]

Mewlo Web Framework Blog Entry #4 - October 14, 2013 - A First-class Addon/Plugin/Extension System

At the end of my last Mewlo blog entry I talked about the central role of an addon/plugin/extension system in Mewlo, and the idea that it would be  a natural first structure to start coding.

There are very good reasons to start with an addon/plugin/extension system.  First and foremost because it makes it easier for other coders to start independently writing drop-in code packages, and makes it easier to keep these updated and integrated.  But I think another key benefit is that it brings into sharp relief early in development the need to craft an API to interact with the rest of the system.  The earlier we force ourselves to view the system as an API, the better.

What exactly should the addon/plugin/extension system for a web framework do? What features does it need to have?

Let's start by restating briefly the main goal of a (web) framework.  The goal is to provide a base upon which custom coding can be done.  A framework enables coders to avoid re-writing the same structures and code for each new project, and provides a well-organized and easy-to-maintain structure that can be extended.

So it should come as no surprise that all non-trivial frameworks have facilities for adding and integrating 3rd party extensions.

For web frameworks, we can distinguish two aspects of how such extensions work:

First, we can look at the actual API provided for interacting with extensions, and how they communicate with the rest of the framework.  Here we find generally quite robust implementations.

Generally one sees a signaling system, where components can register to receive signals when certain events occur, and can register their own signals that listeners can subscribe to.  A signaling system is a key way to let modules interact without requiring them to know too much about each other.

One also often sees a kind of component "registry", where extensions can add new objects that can be used by other code, or new functions that can be called, or where extensions can replace existing components.

However, while the internal API for facilitating interactions between extensions and the core system is often robust and clean, web frameworks generally provide only the most rudimentary higher-level maintenance support for managing extensions.

What exactly do I mean by "higher-level maintenance support for managing extensions"?

If we take a look at software that is one-level higher-up and removed from a web framework, like a Content Management System (CMS) or a blog, what we often see is quite elaborate support system for maintaining extensions.

Such a support system can include an administrative backend that can provide information about current installed extensions and their version and status, and a button to click to check for extension updates online.  The online update check can be as full-featured as a desktop update check tool, providing information about new versions available -- the changes and release dates, etc.  The system may also support dependency checking between multiple modules, and report on incompatibilities and dependencies between extensions.  The administration system may even support searching for extensions online, and then automatically downloading and installing (or uninstalling).

In one aspect, extension updates/installs/uninstalls for online web services are more complicated then desktop applications, and that is when the database needs to be substantially altered.  Whether one is installing, updating, or uninstalling, the need to perform database upgrades can be tricky (especially for a live system).  Frequently one may want to save a database backup to be able to roll-back a problematic update, and one may need to take the system offline while updating.

I take the position that such functionality is exactly the kind of thing that a framework should do more to assist: It is common functionality found in most modern web applications, represents a substantial amount of coding, is hard to code properly and is hard to maintain.  This is especially true for the situations where an extension update needs to perform some one time database updates.  In short, this is an areas that a web framework should do much more to assist with -- building a web application should not require the coder to reinvent the wheel when it comes to supporting extensions.  And yet it is an area that few (any?) web frameworks support robustly.

The support for such extension management and updating is a good example of how Mewlo is trying to occupy a middle ground between traditional web frameworks and higher level content management systems.  In some ways you might say that Mewlo is more of a Content Management System Construction Kit.

[mouser]

By the way, for those interested in being part of the project, I'm planning on holding the First Annual Mewlo Conference in Champaign Illinois in May 2014.

[mouser]

FYI: I'm just getting back into Mewlo work after a couple of months break.

[mouser]

Mewlo Web Framework Blog Entry #5 - January 29, 2014 - Third Party Components

So I took a little time off working on Mewlo, but I'm back to it now, and starting to have a little more fun with it.

Today I thought I'd talk a little bit about which components of Mewlo will be handled (at least initially) by 3rd party open source libraries.

I see Mewlo occupying a middle ground between Django, a framework which uses very few 3rd party components, and Pylons/Pyramid, which is heavily reliant on 3rd party components (both excellent frameworks).

First, let's talk about different ways one might utilize 3rd party components/libraries in a framework.

In the Pylons/Pyramid framework, the framework works hard to make it possible for programmers to use alternative components for major core functions -- such as the database engine, the template engine, etc.  The advantages of such an approach is that the coder is free to use whichever alternative components they prefer.  The disadvantages are that the framework cannot rely on the existence of features and functions, and must sometimes go to extreme lengths to coordinate actions, resulting in overly-complex code.  Sometimes operations must be left to the coder to implement simply because the framework cannot know how the operations would be implemented by different components.  Each component has it's own way of working and it's own coding conventions.

Django, in using custom in-house components for most things (database engine, templating system), can offer a more uniform API and a cleaner more integrated framework.  Components are highly optimized for the framework and follow a unified set of conventions.  The built-in Django components are actually quite excellent.  The disadvantages of such an approach include the in-house coding and maintenance effort required for such components, rather than relying on a high quality active third party open source component, and inflexibility of not allowing programmers to choose their own components for core features (database engine, templates, etc.).

With Mewlo, a conscious effort has been made to follow a philosophy of "one right way to do things".  As such, we are not concerned with offering programmers the flexibility of using their own database engine, template engine, etc.  I realize this can be a controversial decision for some -- and where there is no real complexity cost associated with offering choices (as is the case with template engines), we may relax this strategy.  But in general, we begin with the premise that enforcing the use of a specific component set is not a bad thing.

Furthermore, Mewlo leans much more heavily in the direction of reinventing-the-wheel and writing code in-house, for most functionality.  As such we avoid the use of components when such components are integrated tightly into the framework -- it's my belief that the more different components maintained by different organizations, the more moving parts there are that lead to messier and convoluted code.

Having said that, Mewlo does use some key 3rd party components/libraries.  These components represent a substantial amount of non-trivial intricate code and they are actively developed and maintained by other entities.  It is judged that reinventing these components would not be a good expenditure of time, and we have tried to focus on components that do one thing well.

But, it should be noted, that Mewlo makes an attempt to abstract the interface to these components using thin-wrappers, so that all user-written code goes through a Mewlo API, rather than interacting with these 3rd party components directly.  This has several benefits.  First, it allows Mewlo to present a uniform outwardly-facing API interface.  Second, it allows us to replace components in the future without changing the basic API.  Third, it's consistent with our strategy of designing Mewlo as an language-neutral OOP hierarchy that we could move to another language.

Note that the 3rd party Components I am speaking of here are python libraries that play essential core roles in Mewlo.  I am not talking about occasions when some custom user code will want to use an arbitrary library to perform some limited action.


Template Engine - Jinja2
------------------------

We have preliminarily chosen the Jinja2 templating system for Mewlo (we could have just as easily chosen mako).  As discussed earlier -- a programmer using Mewlo does not invoke Jinja2 functions directly, but acts through an abstract Mewlo class that could easily be extended to support additional templating engines.


ORM Database Layer - SqlAlchemy
-------------------------------

For database operations that do not have to run at the absolutely fastest speed, there are huge advantages to using an object-oriented ORM database layer.  An ORM database library/component is an intricate thing with lots of moving parts that relies on some esoteric language features.  As such, it's exactly the kind of thing that we would prefer to hand off to a well-maintained 3rd party component.  SqlAlchemy fits this bill nicely.  Again, all database operations are done via Mewlo classes, which abstract the calls to SqlAlchemy and facilitate DRY coding with minimal reuse of configuration information.


Form Construction and Validation - WTForms
------------------------------------------

Mewlo currently uses the small WTForms library to handle form creation, presentation, and validation.  Form handling is small enough in scope, that we could replace this in the future with our own custom library -- or move to a larger form library.  For now it represents a good solid minimal form handling component.


Low-level Web Objects - Werkzeug
--------------------------------

Mewlo currently uses the Werkzeug library to handle the low level request and response objects (including cookies, etc.) that lie at the heart of a web framework.  The choice of Werkzeug does violate one of our guidelines for choosing 3rd party components in that it includes code to provide a large number of features that we have no interest in using.  Because we only use minimal pieces of the Werkzeug library, it is highly likely that we will rewrite this component in-house for Mewlo, or move to a much more minimal library for request and response handling.


Others
------

There are several parts of Mewlo that have not been implemented yet, but which will most likely depend on the adoption of 3rd party component libraries:

    * Css/Javascript/Ajax frameworks
    * Internationalization library