topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Friday March 29, 2024, 1:19 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Fabricate: the better "make". Finds dependencies automatically for any language  (Read 10577 times)

berwyn

  • Honorary Member
  • Joined in 2007
  • **
  • default avatar
  • Posts: 8
    • View Profile
    • Donate to Member
fabricate is a build tool that finds dependencies automatically for any language. It's small and just works. No hidden stuff behind your back. It was inspired by Bill McCloskey's "make" replacement, memoize, but fabricate works on Windows as well as Linux.

Features
  •     Never have to list dependencies.
  •     Never have to specify cleanup rules.
  •     The tool is a single Python file.
  •     It uses MD5 (not timestamps) to check inputs and outputs.
  •     You can learn it all in about 10 minutes.
  •     You can still read your build scripts 3 months later.

Show me an example!

Code: Python [Select]
  1. from fabricate import *
  2.  
  3. sources = ['program', 'util']
  4.  
  5. def build():
  6.     compile()
  7.     link()
  8.  
  9. def compile():
  10.     for source in sources:
  11.         run('gcc -c ' + source + '.c')
  12.  
  13. def link():
  14.     objects = ' '.join(s + '.o' for s in sources)
  15.     run('gcc -o program ' + objects)
  16.  
  17. def clean():
  18.     autoclean()
  19.  
  20. main()

Continued on google code ...

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
syntax looks nice.
would be interesting to read a comparison of the various make replacements.

jgpaiva

  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 4,727
    • View Profile
    • Donate to Member
That looks interesting..
But is there any way for it to parallelize the build? (like 'make -j 4' runs 4 concurrent processes?)

berwyn

  • Honorary Member
  • Joined in 2007
  • **
  • default avatar
  • Posts: 8
    • View Profile
    • Donate to Member
No, there is currently no way to parallelize the build.  There was quite a discussion on fabricate's google group on the subject, and I believe a reasonable solution was proposed.  You could ask there to see how far it got.

jgpaiva

  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 4,727
    • View Profile
    • Donate to Member
Oh, that's sort of a killer for me, it makes a big difference to compile using 5 threads in a quadcore :)
But I really like the syntax of fabricate. I went through hell to write a makefile with multiple targets and build directories for my final project, I bet it'd been easier with fabricate!

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
You could also go for SCons? Solid enough that id software uses it, supports parallelization, is super flexible (written in Python), and not that hard to use :)
- carpe noctem

jgpaiva

  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 4,727
    • View Profile
    • Donate to Member
SCons looks interesting. Have you tried it? The fact that it's python (notice that fabricate also is) is an advantage, I'm interested in learning it eventually.
I've looked at the documentation, looks fairly similar to fabricate (I suppose it's actually the other way around), and has support for parallelization (even though I didn't find the help section on it, but it does have the '-j' command line trigger :) )

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Yeah, I've used SCons for a while - never did anything big with it though, and put it on hold for a while because it didn't pick up VS2008 paths by default (and didn't feel like hacking up my construction files). But it's a good tool, and I plan on taking it up again :)
- carpe noctem

berwyn

  • Honorary Member
  • Joined in 2007
  • **
  • default avatar
  • Posts: 8
    • View Profile
    • Donate to Member
Yes, I've tried SCons.  It looked good at first, but in the end, being so frustrated by not being able to do things with it easily, was what drove us to write fabricate.  But try it ... if you find that you are not frustrated by SCons in the end, I'd like to hear from you.

jgpaiva

  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 4,727
    • View Profile
    • Donate to Member
Oh, I didn't understand you were actually the author of fabricate! That's cool ;)

Yep, the next time I need to build something big, I sure will consider these alternatives.

ewemoa

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 2,922
    • View Profile
    • Donate to Member
Thanks for posting about this.

I've started migrating a couple of small projects from custom windows/msdos batch scripts to using fabricate.  (Perhaps I should give SCons a try at some point, but I'm all for starting out with simpler (meant in a good way here!) options first.)

I'm not sure if I'm making good use of it, but for FWIW, I'm finding it much easier to use Python (compared to using batch scripts) to manage building exes, preparing an archive file for distribution, running tests, etc.

For reference, the kinds of things this has helped in gluing together include:

  ahk2exe.exe
  7z.exe
  jsl.exe
  mkdir
  copy
  rmdir

I thought I'd mention some things I found useful:

  using shell() w/ shell=True seemed to make certain things function better
  using shell() w/ silent=False was nice as it allowed getting output from tests
  use '*list' in calls to shell() to handle command lines of the form:

    this.exe file1.js ... fileN.js

  where list is:

    ['file1.js', ..., 'fileN.js']

  the call to shell() would look like:

    shell('this.exe', *list)

  which IIUC is similar to:

    shell('this.exe', 'file1.js', ..., 'fileN.js')

Nice to have source to discover these things (especially when my doc-reading skills don't succeed) :)

Anyway, thanks again!
« Last Edit: March 12, 2010, 02:27 AM by ewemoa »

berwyn

  • Honorary Member
  • Joined in 2007
  • **
  • default avatar
  • Posts: 8
    • View Profile
    • Donate to Member
Yep, there are no docs as thorough as the source ;)!

You're right that the shell command is handy, and probably warrants better documentation.  You can, of course, get docs on it without browsing the source with python's help command: help(shell).  But now that I look at it again, that tail doc sure is obscure.  It could do with improvement.

If you want to write up a suggested doc for me on shell(), I could see about getting it put onto the fabricate website.

CWuestefeld

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 1,009
    • View Profile
    • Donate to Member
fabricate[/b] is a build tool that finds dependencies automatically for any language

How does that work? It seems rather magical to me.

ewemoa

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 2,922
    • View Profile
    • Donate to Member
How does that work? It seems rather magical to me.

Some excerpted text:

fabricate lets you simply type run('command', 'arg1', 'arg2'), and it will run the command and figure out its dependencies (and outputs) automatically. Next time it'll only actually execute the command if those dependencies have changed (or the outputs have been modified or don't exist).

It finds dependencies by one of two methods:

    * strace: fabricate first tries to use the Linux strace command to log system calls to open() and determines what files were read and modified that way. This is the original method used by Bill McCloskey's memoize.
    * File access times (atimes): Some systems (Windows) don't have strace, and in that case fabricate looks at the atimes of files before and after the command was run, and from those can figure out which files each command accesses. It also uses file modification times (mtimes) to determine the command's output files. It's fast and simple, and it works on NTFS, FAT32 (thanks to some tricks), as well as Linux file systems that have atimes enabled.

I found the above text at the HowItWorks page.  There is further text there including a warning for use on Windows systems :)

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
...atime is disabled by default on Win7, and power users tend to disable it on previous systems. It's a mostly useless feature that slows things down without bringing much benefit. Using strace or atime to figure out dependencies sounds pretty flaky to me at any rate.
- carpe noctem

berwyn

  • Honorary Member
  • Joined in 2007
  • **
  • default avatar
  • Posts: 8
    • View Profile
    • Donate to Member
Yep, as you say, atimes is indeed flaky -- as has been observed on the fabricate forum.  Also, as you say, atimes they aren't enabled by default on windows 7.  We really need somebody to implement an strace-like solution for windows.  It is quite possible using the EasyHooks library or other methods.  But it takes some work as several windows API calls need to be hooked: open/read/write, create process, and change directory.

For now, atimes fill our need, but at some point the pitfalls will probably get the better of us and we'll need to write the EasyHooks solution.  Or someone else will get there first.  It's a good, clean, hacker project.  If someone feels keen to follow this path, I'd be interested to hear from them.