Fabricate: the better "make". Finds dependencies automatically for any language

[berwyn]

fabricate is a build tool that finds dependencies automatically for any language. It's small and just works. No hidden stuff behind your back. It was inspired by Bill McCloskey's "make" replacement, memoize, but fabricate works on Windows as well as Linux.

Features

•     Never have to list dependencies.
•     Never have to specify cleanup rules.
•     The tool is a single Python file.
•     It uses MD5 (not timestamps) to check inputs and outputs.
•     You can learn it all in about 10 minutes.
•     You can still read your build scripts 3 months later.

Show me an example!

Code: Python [Select]

1. from fabricate import *
2.  
3. sources = ['program', 'util']
4.  
5. def build():
6.     compile()
7.     link()
8.  
9. def compile():
10.     for source in sources:
11.         run('gcc -c ' + source + '.c')
12.  
13. def link():
14.     objects = ' '.join(s + '.o' for s in sources)
15.     run('gcc -o program ' + objects)
16.  
17. def clean():
18.     autoclean()
19.  
20. main()

Continued on google code ...

[mouser]

syntax looks nice.
would be interesting to read a comparison of the various make replacements.

[jgpaiva]

That looks interesting..
But is there any way for it to parallelize the build? (like 'make -j 4' runs 4 concurrent processes?)

[berwyn]

No, there is currently no way to parallelize the build.  There was quite a discussion on fabricate's google group on the subject, and I believe a reasonable solution was proposed.  You could ask there to see how far it got.

[jgpaiva]

Oh, that's sort of a killer for me, it makes a big difference to compile using 5 threads in a quadcore
But I really like the syntax of fabricate. I went through hell to write a makefile with multiple targets and build directories for my final project, I bet it'd been easier with fabricate!

[f0dder]

You could also go for SCons? Solid enough that id software uses it, supports parallelization, is super flexible (written in Python), and not that hard to use

[jgpaiva]

SCons looks interesting. Have you tried it? The fact that it's python (notice that fabricate also is) is an advantage, I'm interested in learning it eventually.
I've looked at the documentation, looks fairly similar to fabricate (I suppose it's actually the other way around), and has support for parallelization (even though I didn't find the help section on it, but it does have the '-j' command line trigger )

[f0dder]

Yeah, I've used SCons for a while - never did anything big with it though, and put it on hold for a while because it didn't pick up VS2008 paths by default (and didn't feel like hacking up my construction files). But it's a good tool, and I plan on taking it up again

[berwyn]

Yes, I've tried SCons.  It looked good at first, but in the end, being so frustrated by not being able to do things with it easily, was what drove us to write fabricate.  But try it ... if you find that you are not frustrated by SCons in the end, I'd like to hear from you.

[jgpaiva]

Oh, I didn't understand you were actually the author of fabricate! That's cool

Yep, the next time I need to build something big, I sure will consider these alternatives.

[ewemoa]

Thanks for posting about this.

I've started migrating a couple of small projects from custom windows/msdos batch scripts to using fabricate.  (Perhaps I should give SCons a try at some point, but I'm all for starting out with simpler (meant in a good way here!) options first.)

I'm not sure if I'm making good use of it, but for FWIW, I'm finding it much easier to use Python (compared to using batch scripts) to manage building exes, preparing an archive file for distribution, running tests, etc.

For reference, the kinds of things this has helped in gluing together include:

  ahk2exe.exe
  7z.exe
  jsl.exe
  mkdir
  copy
  rmdir

I thought I'd mention some things I found useful:

  using shell() w/ shell=True seemed to make certain things function better
  using shell() w/ silent=False was nice as it allowed getting output from tests
  use '*list' in calls to shell() to handle command lines of the form:

    this.exe file1.js ... fileN.js

  where list is:

    ['file1.js', ..., 'fileN.js']

  the call to shell() would look like:

    shell('this.exe', *list)

  which IIUC is similar to:

    shell('this.exe', 'file1.js', ..., 'fileN.js')

Nice to have source to discover these things (especially when my doc-reading skills don't succeed)

Anyway, thanks again!

[berwyn]

Yep, there are no docs as thorough as the source !

You're right that the shell command is handy, and probably warrants better documentation.  You can, of course, get docs on it without browsing the source with python's help command: help(shell).  But now that I look at it again, that tail doc sure is obscure.  It could do with improvement.

If you want to write up a suggested doc for me on shell(), I could see about getting it put onto the fabricate website.

[CWuestefeld]

fabricate[/b] is a build tool that finds dependencies automatically for any language

-berwyn (July 27, 2009, 05:24 AM)

How does that work? It seems rather magical to me.

[ewemoa]

How does that work? It seems rather magical to me.

-CWuestefeld (March 12, 2010, 12:06 PM)

Some excerpted text:

fabricate lets you simply type run('command', 'arg1', 'arg2'), and it will run the command and figure out its dependencies (and outputs) automatically. Next time it'll only actually execute the command if those dependencies have changed (or the outputs have been modified or don't exist).

It finds dependencies by one of two methods:

    * strace: fabricate first tries to use the Linux strace command to log system calls to open() and determines what files were read and modified that way. This is the original method used by Bill McCloskey's memoize.
    * File access times (atimes): Some systems (Windows) don't have strace, and in that case fabricate looks at the atimes of files before and after the command was run, and from those can figure out which files each command accesses. It also uses file modification times (mtimes) to determine the command's output files. It's fast and simple, and it works on NTFS, FAT32 (thanks to some tricks), as well as Linux file systems that have atimes enabled.

I found the above text at the HowItWorks page.  There is further text there including a warning for use on Windows systems

[f0dder]

...atime is disabled by default on Win7, and power users tend to disable it on previous systems. It's a mostly useless feature that slows things down without bringing much benefit. Using strace or atime to figure out dependencies sounds pretty flaky to me at any rate.

[berwyn]

Yep, as you say, atimes is indeed flaky -- as has been observed on the fabricate forum.  Also, as you say, atimes they aren't enabled by default on windows 7.  We really need somebody to implement an strace-like solution for windows.  It is quite possible using the EasyHooks library or other methods.  But it takes some work as several windows API calls need to be hooked: open/read/write, create process, and change directory.

For now, atimes fill our need, but at some point the pitfalls will probably get the better of us and we'll need to write the EasyHooks solution.  Or someone else will get there first.  It's a good, clean, hacker project.  If someone feels keen to follow this path, I'd be interested to hear from them.