Agavi

Fixing SVN Merge History in Git Repositories

Wed, 01 Feb 2012 15:22:00 +0000

Assuming you use the fabulous svn2git to convert your SVN repository to a Git one, you might still run into a problem: merge history.

Why And How To Fix Merge History

Things Might Be Just Fine

If you have always used SVN 1.5 or newer and thus have svn:mergeinfo properties for all your merges, you can ignore this part, as git-svn will correctly identify merges from, say, a branch back to trunk (or master once the conversion is done) and attach two parents to that commit: the previous commit in master, and the last commit from the branch that’s being merged.

In such a case, your history will look something like this:

$ git log --pretty=format:\"%h %ad | %s%d [%an]\" --graph --date=short
* 0d34dbe 2009-08-12 | allow passing of arbitrary arguments to ant (HEAD, master) [felix]
*   a199e71 2009-07-06 | merging the changes from [3853:4186/branches/felix-tutorial-app-rewrite] major rewrite of tutorial apps major rewrite of build scripts [felix]
|\  
* | 2121341 2009-07-06 | removing staging apps for backmerge, somehow svn mucks up if I don't [felix]
| *   145e1aa 2009-07-06 | merging the changes from [4165:4184/branches/felix-build-scripts-rewrite] fixes #1115 (felix-tutorial-app-rewrite) [felix]
| |\  
| | * 2a9b5cc 2009-07-06 | fix excludes for svn directories (felix-build-scripts-rewrite) [felix]
| | * ec374ec 2009-07-02 | reorganized the stylesheets a little, make a custom base that does not set doctypes on the output and a stylesheet for the website as well as for the xhtml documentation that inherit from it [felix]
| | * f119381 2009-07-01 | modified the toolkits build system to allow overriding the xsl for the toc-generation [felix]
| | * 576b71b 2009-06-29 | creating branch for build scripts rewrite, fixing build problems in dita-ot 1.4.2 and greater refs #1115 [felix]
| |/  
| * 6e8e682 2009-06-26 | added a note about Action::executeRead() to the chapter about module creation fixes #1091 [felix]
| * 976c384 2009-02-18 | creating a branch for the rewrite of the tutorial apps to incorporate latest best practices. [felix]
* | 2121341 2009-07-06 | removing staging apps for backmerge, somehow svn mucks up if I don't [felix]
|/  
* fd87653 2008-10-31 | dump current versions of refguide [mikeseth]

But You’re Probably Screwed

If, however, you did not have merge tracking yet, the history will look like this:

$ git log --pretty=format:\"%h %ad | %s%d [%an]\" --graph --date=short
* 0d34dbe 2009-08-12 | allow passing of arbitrary arguments to ant (HEAD, master) [felix]
* a199e71 2009-07-06 | merging the changes from [3853:4186/branches/felix-tutorial-app-rewrite] major rewrite of tutorial apps major rewrite of build scripts [felix]
* 2121341 2009-07-06 | removing staging apps for backmerge, somehow svn mucks up if I don't [felix]
* fd87653 2008-10-31 | dump current versions of refguide [mikeseth]

The reason becomes apparent when we use git-rev-list:

$ git rev-list --all --graph --oneline --date-order
* 0d34dbe allow passing of arbitrary arguments to ant
* a199e71 merging the changes from [3853:4186/branches/felix-tutorial-app-rewrite] major rewrite of tutorial apps major rewrite of build scripts
* 2121341 removing staging apps for backmerge, somehow svn mucks up if I don't
| * 145e1aa merging the changes from [4165:4184/branches/felix-build-scripts-rewrite] fixes #1115
| | * 2a9b5cc fix excludes for svn directories
| | * ec374ec reorganized the stylesheets a little, make a custom base that does not set doctypes on the output and a stylesheet for the website as well as for the xhtml documentation that inherit from it
| | * f119381 modified the toolkits build system to allow overriding the xsl for the toc-generation
| | * 576b71b creating branch for build scripts rewrite, fixing build problems in dita-ot 1.4.2 and greater refs #1115
| |/  
| * 6e8e682 added a note about Action::executeRead() to the chapter about module creation fixes #1091
| * 976c384 creating a branch for the rewrite of the tutorial apps to incorporate latest best practices.
|/  
* fd87653 dump current versions of refguide

The merge commits (145e1aa and a199e71) each have only one parent and do not point to the merged commits. This ancestry is then lost when running git log. As a result, it will later be really difficult or impossible to find the commits from those branches again, especially if the corresponding branches have been removed from the origin.

Grafts To The Rescue

You can override parents of a commit in Git using a file called grafts, which needs to reside in .git/info/. In consists of several lines with commit hashes separated by a space character, where the first hash is the commit you want to define parents for, and the following hashes are the parents.

Note: you must use full hashes, not abbreviated ones.

To fix the history above, we must define the parents in .git/info/grafts:

a199e7194c5a4382c7d1057c84a55fcb535c8ed2 2121341005ef4fee0c1755d9ee083b6f0fbaf295 145e1aa687c7b1277c683b8116324a1552ba6454
145e1aa687c7b1277c683b8116324a1552ba6454 6e8e6828f7be352571faccef981568f4acc9ccb1 2a9b5ccf4b0870973508b37bef8d7759889a4f27

Note: for a merge from, say, master to a branch (to sync the branch with master/trunk), you would use $synccommit $lastmastercommit $lastbranchcommit.

If we now run git log --pretty=format:\"%h %ad | %s%d [%an]\" --graph --date=short, things will look just like we expected.

We can then run git filter-branch to make the grafts permanent (see step “Manual Conversion” below). Note that this will change commit hashes.

Manual Conversion

Run svn2git http://svn.example.org/repos --no-minimize-url --metadata --authors authors.txt
Create a file .git/info/grafts (you can use git svn find-rev r$rev $branch to find a Git commit hash for an SVN revision number)
Run git filter-branch --tag-name-filter cat -- --all to make the grafts permanent
Check if the history looks fine
Run git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d to clean up the backups created by git-filter-branch
Remove the now obsolete .git/info/grafts

Automatic Conversion

For the Agavi migration, we wrote a script to automate this process; it uses a grafts file that contains SVN revision numbers (and their branch names) and converts it to a grafts file fit for consumption by Git. We will put that script up on GitHub together with the rest of our migration toolkit in a few days.

Broken Branches

The Agavi documentation repository contains a broken branch, where the branch was made from only a portion of trunk, not from the whole of trunk, in revision 4285. When running this through svn2git, the whole history would be terribly confused and a lot of commits would appear as duplicates since git-svn interpreted the branch point wrong.

The fix in this case was to simply ignore that branch and only use its merge back to trunk (in revision 4308), as the loss of history information in this case was negligible. We told svn2git top stop at the revision before the branch was created, and resume at the revision where the branch gets merged back to trunk (this required a patch to svn2git):

$ svn2git http://svn.agavi.org/documentation --revision 2721:4284 --no-minimize-url --metadata --authors authors.txt
$ git svn fetch -q --revision 4308:HEAD
$ git merge -q svn/trunk

We’re only merging svn/trunk in this case as all the commits from 4308 onwards were made to trunk. Also, between 4284 and 4308, all commits in this repository were in the branch we’re skipping - had there been commits to trunk or other branches, things would have been a tiny bit more complicated, with several calls to git svn fetch and git merge to put the pieces together correctly.

Finding Problems in Commit Messages and Tickets When Migrating to GitHub

Sat, 21 Jan 2012 14:56:08 +0000

Sometimes, you commit a change, but then you realize that you did not mention the ticket ID in your commit message. Or, worse: you use the wrong ticket ID.

The usual remedy is adding this info to the ticket by hand. On the Agavi Trac, when a ticket ID was incorrect in a commit message, we would typically remove the corresponding entry from the wrong ticket and move it to the correct one.

When migrating to GitHub and replaying all commits and ticket changes, that becomes a problem of course: the commit message still contains text like “fixes #123” when in fact it was supposed to reference ticket #124.

It is therefore necessary to programmatically find and mitigate such problem cases and addressing them before importing things on GitHub. Failure to correct just a single of these problems may break the whole import by having tickets remain open randomly even though they were supposed to be closed and so forth. Here is a list of situations that need to be addressed:

Commit Messages Not Showing Up in Their Tickets

Sometimes a reference to a ticket ID in a commit message does not show up in the corresponding ticket. There are several reasons why this may happen:

The message was never supposed to reference a ticket (“Attempt #2 at this stupid merge”) - not a problem in Trac since there is no “refs” or other keyword in front of “#2”, but on GitHub, this commit would show up in ticket #2.
=> Such a commit message must be modified to prevent this situation.
The message contains the wrong ticket ID and was removed from that wrong ticket (and inserted by hand on the correct ticket)
=> Such a commit message must be modified to prevent this situation.
The Trac post-commit hook was not installed or malfunctioning
=> This usually does not require any action.

Differences in Behavior Between Trac and GitHub

Aside from the first point above, there are other subtle but important differences between how Trac and GitHub parse commit messages:

A commit closes several tickets at once without repeating the keyword, e.g. “Add bacon, closes #4 and #5”. This will close both tickets in Trac, but on GitHub, only ticket #4 is closed while ticket #5 is referenced.
=> The commit message must be changed so a “closes” or “fixes” keyword appears before every ticket ID.
A commit shows up just fine in the Trac ticket, but the reference would not be detected by GitHub, e.g. due to a missing space character (“This closes #5for real” would close #5 in Trac but do nothing on GitHub).
=> A space character must be inserted
A commit closes a ticket that was already closed (this will cause the commit to not show up at all on GitHub).
=> The commit message must be changed, or a reopen command must be sent to the ticket before pushing the second commit.

Human Interference

People do stupid things in their ticket database, so aside from the situation we already discussed where a commit referenced the wrong ticket number, we’ll have to watch out for:

A commit referencing a ticket that, at the time the commit was made, did not exist yet (a mistake frequently made by time traveling contributors making commits from the future :>).
=> The easiest fix here is to simply make sure the ticket is created before the commit is pushed.
A commit message that shows up in the right (or wrong) ticket, but has been altered (for whatever reason and by whatever means).
=> The commit message must be altered accordingly (or not if that’s not desirable). Optionally, punch the person who fiddled with the message.

Largely Cosmetic Issues

Finally, a few things in commit messages or tickets may not be a problem for the import, but could still be nice to fix:

Merges done using svnmerge have the nasty habit of repeating all merged commit messages and thus cluttering the history.
=> It might be nice to remove the clutter from the commit message so all the tickets mentioned in the merged revisions remain clean.
A ticket was closed manually, but with a reference to the corresponding commit.
=> This could be fixed by putting the ticket ID into the commit message, which will take advantage of GitHub’s automatic referencing or closing of tickets in such a case.

Basic Preparations for GitHub Ticket Migration

Tue, 03 Jan 2012 07:00:00 +0000

Before modifying the ticket and comment contents themselves (to convert them from TracWiki syntax to Github Flavored Markdown), a few more fundamental changes are necessary:

Find missing ticket IDs (the Agavi Trac has 83 “gaps” in the ticket ID sequence; those were all spam tickets that got deleted, but not in time for a legit ticket showing up with a later ID in the database)
Compose a list of all ticket reporters, ticket change authors (e.g. people leaving a comment) and attachment authors and create a mapping to their GitHub user names or plain display names that will be used during import
Migrate all milestones to the GitHub issue tracker
Migrate all versions (as labels) to the GitHub issue tracker
Migrate all components (as labels) to the GitHub issue tracker
Migrate all ticket types (as labels) to the GitHub issue tracker
Do something about Priorities and Severities

1. Finding Missing Ticket IDs

That one is fairly easy:

$ sqlite3 trac.db 'SELECT id FROM ticket;' | php -r '
$ids = file("php://stdin", FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
echo implode("\n", array_diff(range(1, max($ids)), $ids)) . "\n";
'

The resulting list needs to be stored somewhere for later, when we replay all ticket changes - that script will produce empty dummy tickets for the missing IDs to make sure all ticket IDs line up between Trac and GitHub.

2. List Ticket Reporters/Authors

We only need SQL for this:

sqlite3 trac.db '
SELECT DISTINCT (user || " = ") FROM (
    SELECT reporter AS user FROM ticket
    UNION ALL
    SELECT author AS user FROM ticket_change
    UNION ALL
    SELECT author AS user FROM attachment
) ORDER BY LOWER(user) ASC;
'

The result can be put into a file that is then filled by hand with references to people. In our case, we chose to use either a GitHub username (if known, requires some research in certain cases), half of the person’s e-mail address, or a nickname in quotes:

anonymous = "anonymous"
david = @dzuelke
steve@REDACTED.com = steve@…

“REDACTED.com” is obviously not what was really in there; the ellipsis (“…”) is correct however.

This map will be used by a script later when replaying ticket changes via the GitHub API in order to embed information about the original authors. By using GitHub usernames with the “@someone” notation, a link will be created; we just have to get things right on the first attempt or some people will complain about notification spam ;)

3. Migrate Milestones

Very straightforward as the GitHub issue tracker supports milestones. They existing milestones in the Trac database are simply ported over to GitHub; milestone “AgaviForge” will be dropped however (and the corresponding tickets later reassigned).

4. Migrate Versions (as Labels)

Versions as registered in Trac will simply be imported as labels like “0.10.2” or “1.0.8”. For now, we will import all versions except for “HEAD”, “HEAD-1.0” and “HEAD-0.11” since those only cause clutter and thus far have rarely been used.

This will create a considerable amount of labels just for the versions; we might eventually merge them all into series labels (like “1.0.x” or “1.1-latest”, similar to e.g. Ruby on Rails). When doing that, we might also drop old series labels that are not maintained anymore, simply to cut down on clutter.

One last thing to decide upon of course is the color codes for the versions. No great ideas yet, as the colors are shared with components and ticket types (see below).

5. Migrate Components (as Labels)

Components will be migrated to labels with identical names, with the exception of “documentation” (will be moved to a separate repository), “website” (never has been part of the SVN repos, but will be moved to its own GitHub repository eventually) and “OTHER” (no label will be assigned to such issues).

Again, color coding is something that’s left to be decided (suggestions welcome).

6. Migrate Types (as Labels)

There currently are just three issue types: “defect”, “enhancement” and “task”. Those will be migrated as-is, with defects getting a flashy red color, enhancements a nice green, and tasks… well… turquoise maybe?

7. Priorities and Severities

Priorities will simply be dropped, they have never proven to be useful. Same for severities, although we might have a dedicated label for blockers, indicating that resolving such an issue may not be postponed until a later release.

Porting Trac Ticket Attachments to GitHub

Sat, 31 Dec 2011 02:20:32 +0000

Some of the tickets in the Agavi Trac have attachments; these attachment files need special treatment as the GitHub issue tracker itself has no support for attachments.

We’ve created a repository (agavi/trac-ticket-attachments) for this purpose and simply imported the everything from trac/attachments/tickets, which contains just a bunch of folders with the ticket ID and the attachments of the ticket inside them. Some of the files have %20 in them as the uploaded files had spaces in their names; we changed those to actual space characters so the URL escape does not end up in the file name in the Git repository.

We will later rewrite the ticket history so the attachment links point to the corresponding files in this repository.

Testing GitHub Tickets for the Migration

Fri, 30 Dec 2011 21:39:00 +0000

One thing we’ve always put a focus on is having tickets for every change we make, and using these tickets extensively in commit messages and from other tickets so a quick svn blame on a line not only shows a description of the change, but also gives a ticket number that can be looked up on Trac to follow a discussion of the change.

On numerous occasions, I looked at a bit of code, found it to be incredibly stupid, but then did an svn blame anyway to figure out when the change was done. Looking up the ticket on Trac then usually gives a list of all the commits for that change, and maybe even related tickets and discussions. Often enough, that line of code then turned out to not be so stupid after all, and the day was saved.

Good examples are tickets #502 or #1199, which link to related and duplicate tickets, have discussions in them and of course commits references.

All these tickets contain valuable information, so it is vital to preserve all this history. However, the GitHub ticket API does not allow a timestamp in updates, which means that any ticket and any comment will have the date and time of import, and not the old date and time. Commits from the repository, however, will contain the correct timestamps as git2svn preserves that information.

We tested how the GitHub issue tracker behaves with respect to timestamps, and found the following:

For commits referencing a ticket (via “refs #123” or “closes #456”) will use the timestamp of the commit for the reference, not the timestamp of the push
Comments and commit references in tickets are in the order of their creation in the tracker, not in the order of their timestamps
Tickets that do not exist are not updated retroactively when referenced in a commit message

The result is that we can now build a database of commits and ticket changes that are in the order of their timestamps, and replay them step by step, meaning that we will convert the repos, then push a commit, then replay a ticket update or creation, then push another commit and so forth. The result will be that a ticket might contain a comment (with a timestamp of, say, January 2 2012 13:04:12 because that’s the time when our API client replayed that change) followed by a commit reference that closed the ticket (with a timestamp of, say, August 4 2007 08:09:12, because that’s when that change was committed).

Ultimately, this means that we will be able to use GitHub’s native support for “refs #123” or “closes #456”, which will look much nicer than manually having to construct comments through the API about commits referencing the issue.

The remaining issue of course is that any comment and ticket will have the wrong date and time, so we’ll have to include that in the next (along with a notice saying that the ticket/comment was migrated from the old Trac, just so people know), but it simply seems like there is no way around that with the GitHub API.

Regarding severities and priorities, we might drop these altogether along with keywords; components can be done via labels, and so can ticket types (enhancement/task/defect) and versions. Milestones are supported natively by the GitHub issue tracker, so they won’t be a problem.

Certain changes in GitHub, such as changing milestones, labels or assignees, are not displayed in the web interface, even though the changes appear to be recorded internally and are available through the API, so the information itself is not lost.

Moving to GitHub!

Fri, 30 Dec 2011 20:34:00 +0000

We have just begun working on migrating trac.agavi.org and svn.agavi.org to GitHub; there is whole range of things to consider, from referencing users correctly over rewriting commit messages so the correct Git commit references are included to preserving the history of tickets in their right order.

We will add more blog posts as we move along to document the process in case other projects would like to do the same, and also make our migration scripts available if they are sufficiently generic to be useful to others.

Agavi 1.0.7 released!

Thu, 22 Dec 2011 13:59:31 +0000

Agavi 1.0.7 is now available for download at www.agavi.org and through the PEAR channel.

The release contains just two minor changes over RC2, so I’ll simply quote from the final release notes:

The getCredentials() method on an Action is no longer called unconditionally (i.e. whether or not the isSecure() method returned false) but only if the Action is “secure”.

It is now possible to manually call shutdown() on any database adapter to close the underlying connection; another call to getConnection() will cause a reconnect.

AgaviFormPopulationFilter will now populate multiple forms in the order specified in the “populate” request attribute (when populating via an array with form IDs as keys and parameter holders as values) in namespace “org.agavi.filter.FormPopulationFilter”); however, if the value for a key in the “populate” array is boolean true (to re-populate from request data), this form will always be handled first, so error messages are inserted on that form first.

Testing is now compatible with both PHPUnit 3.5 and 3.6. If you want to specify code coverage filters, AgaviTesting::getCodeCoverageFilter() returns the correct instance (singleton or not depending on the PHPUnit version) for you to use. A base constraint class named “AgaviBaseConstraintBecausePhpunitSucksAtBackwardsCompatibility” can be used for constraints that work with both PHPUnit 3.5 and 3.6; if you implement the new matches() method introduced in PHPUnit 3.6 instead of the old evaluate(), it will automatically be called in the proper fashion depending on the PHPUnit version.

AgaviTesting::dispatch() can now call exit() with the appropriate shell status code (the same as returned by a vanilla PHPUnit run) to indicate success or failures/errors to the calling process. This behavior is triggered when the new optional second argument, defaulting to false, is set to true. If set to false, it returns the PHPUnit result object that may be used by custom code to perform further analysis of the test run.

Several other minor changes and fixes are included in this release as well; most notably, AgaviBooleanValidator’s casting and exporting logic has been repaired, and AgaviConsoleRequest now properly creates an AgaviUploadedFile object with STDIN contents (when configured to read those) instead of a plain array. The PHPTAL renderer now supports configuration of character encoding via parameter “encoding”.

The timezone database was updated to version 2011n.

As always, check the CHANGELOG for the full list of enhancements, changes and fixes.

Agavi 1.0.7 RC2 released!

Wed, 14 Dec 2011 20:51:35 +0000

Agavi 1.0.7 RC2 is now available for download at www.agavi.org and through the PEAR channel.

A few minor fixes and changes have made it in since RC1; most notably, PHPUnit 3.6 should work fine now, and the Form Population Filter, when populating several forms (via an array as the value for “populate” with form IDs as keys) now gives precedence to boolean true values (so error messages are inserted on re-populated forms first) and processes forms in the order given in the “populate” array rather than in the order they appear in the document.

Please test this thoroughly and give feedback if necessary so we can roll a final version before the holidays.

Agavi 1.0.7 RC1 released!

Sun, 20 Nov 2011 18:45:54 +0000

Agavi 1.0.7-RC1 is now available for download at www.agavi.org and via the PEAR channel.

As always, the CHANGELOG has the full story, but here is a summary of the most important changes:

The getCredentials() method on an Action is no longer called unconditionally (i.e. whether or not the isSecure() method returned false) but only if the Action is “secure”.
The PHPTAL renderer now supports configuration of character encoding via parameter “encoding”.
It is now possible to manually call shutdown() on any database adapter to close the underlying connection; another call to getConnection() will cause a reconnect.
AgaviTesting::dispatch() can now call exit()? with the appropriate shell status code (the same as returned by a vanilla PHPUnit run) to indicate success or failures/errors to the calling process. This behavior is triggered when the new optional second argument, defaulting tofalse, is set totrue. If set tofalse`, it returns the PHPUnit result object that may be used by custom code to perform further analysis of the test run.
Several other minor changes and fixes are included in this release as well; most notably, AgaviBooleanValidator’s casting and exporting logic has been repaired, and AgaviConsoleRequest now properly creates an AgaviUploadedFile object with STDIN contents (when configured to read those) instead of a plain array.
The timezone database was updated to version 2011n.

Agavi 1.0.6 released!

Sat, 23 Jul 2011 21:43:00 +0100

We’re thrilled to announce that the final version of Agavi 1.0.6 is now available for download at agavi.org and through the PEAR channel.

There were no changes in this release over the release candidate, so please refer to the RC1 release announcement and of course once again the CHANGELOG to see what great new features and changes are included in this new version.

Agavi 1.0.6 RC1 released!

Mon, 11 Jul 2011 12:42:00 +0100

We’re happy to announce that the first release candidate for version 1.0.6 is now available for download at agavi.org and through the PEAR channel.

This release adds a database adapter for Doctrine 2, a renderer for the Twig template engine, a native session storage for ext/sqlsrv and various enhancements such as support for Smarty 3 in the existing renderer and extended configuration abilities for the Doctrine 1 database adapter.

Doctrine 2 support was added, both for the ORM (AgaviDoctrine2ormDatabase) and the DBAL (AgaviDoctrine2dbalDatabase) libraries. The ORM adapter can optionally utilize a pre-configured DBAL connection instead of specifying connection details itself. Both adapters allow for easy extensibility with custom logic by exposing a prepareConfiguration() and a prepareEventManager() method that can be used to customize the configuration and the event manager, respectively. The API documentation for both classes contains a detailed list of all options; a typical ORM adapter configuration would look similar to this:

<database name="master" class="AgaviDoctrine2ormDatabase">
    <ae:parameter name="connection">
        <ae:parameter name="driver">pdo_mysql</ae:parameter>
        <ae:parameter name="host">127.0.0.1</ae:parameter>
        <ae:parameter name="user">root</ae:parameter>
        <ae:parameter name="dbname">test</ae:parameter>
    </ae:parameter>
    <ae:parameter name="configuration">
        <ae:parameter name="metadata_driver_impl_argument">%core.model_dir%/Entities</ae:parameter>
    </ae:parameter>
</database>

An example cli-config.php file demonstrating how to integrate Agavi with the Doctrine command line utilities can be found in etc/database/doctrine2.

The database adapter for Doctrine 1 now allows basic configuration of cache drivers for result and query caches, and added the ability to specify the class to use as the connection event listener - this allows a custom implementation to be used, where more advanced logic can be executed in the preConnect() event handler method, which is useful for modifying the connection or manager objects in ways not supported through configuration options.

It is now possible to use ext/sqlsrv as a session storage via AgaviSqlsrvSessionStorage as an alternative to interfacing with Microsoft SQL Server via PDO. The session table’s data column must be of type “varbinary”.

The Twig template engine is supported through AgaviTwigRenderer; includes and inheritance lookups are performed on the current directory first by default before falling back to the main template directory. The lookup sequence is configurable. The renderer will also attempt to load a localized template from a locale subdirectory first before picking a generic one if the template that initiated the include or extension was a localized template.

AgaviSmartyRenderer received minor adjustments to support Smarty 3 without triggering deprecation notices.

The Sample App now uses the regular “web” context to serve the WSDL for the product service; the WSDL contents can now be retrieved through the URL /products.wsdl.

The timezone database was updated to version 2011h.

A complete list of changes in this release can be found in the CHANGELOG. Please take it for a spin and report any issues before we release a final version soon!

Agavi 1.0.5 released!

Thu, 16 Jun 2011 21:49:09 +0100

The final version of Agavi 1.0.5 is now available at www.agavi.org and through the PEAR channel.

A small number of fixes and changes have made it in since 1.0.5 RC1; the most important one is compatibility with PHPUnit 3.5.

To support PHPUnit 3.5, we had to drop support for PHPUnit 3.4 since a few backwards-incompatible changes were made that we could not work around. Your tests should not be affected, but your run-tests.php will likely need an adjustment to use PHP_CodeCoverage_Filter::getInstance()->addDirectoryToBlacklist() instead of the now-removed PHPUnit_Util_Filter::addDirectoryToFilter().

As always, CHANGELOG and RELEASE_NOTES will be happy to tell you all the details.

Another 1.0.6 release is planned soon with a few minor additions on the road to Agavi 1.1, which has made good progress over the last few weeks.

Agavi 1.0.5 RC1 released!

Mon, 06 Jun 2011 13:06:00 +0100

Agavi 1.0.5 RC1 is now available for download at www.agavi.org and through the PEAR channel.

There are quite a few changes in this release, so I’ll quote from the RELEASE_NOTES:

This release improves the robustness of AgaviFormPopulationFilter especially for XHTML and HTML5, and fixes several issues. The database adapters for ext/mysql, ext/mysqli and ext/pdo (when used with MySQL connections) now also prevent potentially unsafe setting of connection encodings.

AgaviFormPopulationFilter can now recover from various non-fatal parse errors, such as unknown tags or malformed markup in HTML parsing mode, and undefined entities in XML parsing mode. This marks an end to issues with non-XML entities like “ ” in XML parsing mode, and allows the use of HTML5 elements (which libxml does not know yet) in HTML parsing mode (i.e. without using the XML serialization of HTML5). Instead of a boolean true or false, the configuration parameter “ignore_parse_errors” now takes the possible values “LIBXML_ERR_NONE” (equivalent to boolean false), “LIBXML_ERR_WARNING” (quite useless), “LIBXML_ERR_ERROR” (the new default) and “LIBXML_ERR_FATAL” (equivalent to boolean true). FPF will suppress errors and continue operation if the errors encountered during parsing do not exceed the configured maximum ignore level. In the event of a fatal error, such as malformed markup in XML parsing mode, FPF will silently abort execution if “LIBXML_ERR_FATAL” is configured, as fatal errors are not recoverable. In all other cases, FPF will throw an exception or recover from the error, depending on the configured ignore level and the highest error level encountered during parsing. The new default ignore level “LIBXML_ERR_ERROR” means that users will see significantly fewer errors during development; it also means, however, that in HTML parsing mode in particular, well-formedness errors may remain unnoticed. For this reason, FPF will continue to log errors (if logging is enabled), and map the error level of the parse error (LIBXML_ERR_WARNING, LIBXML_ERR_ERROR or LIBXML_ERR_FATAL) to the error level of the logging system (AgaviILogger::WARN, AgaviILogger::ERROR or AgaviILogger::FATAL). As a consequence, the “logging_severity” configuration parameter has been removed.

AgaviZendclouddocumentserviceDatabase is a database adapter for Zend Framework’s Zend_Cloud_DocumentService. It has a convenience interface mirroring most of the Zend_Cloud_DocumentService_Adapter functions in such a way that the collection/domain/database name does not have to be passed to every call. The collection name can then be configured in databases.xml.

AgaviMysqlDatabase, AgaviMysqliDatabase and AgaviPdoDatabase now throw errors when attempting to use a statement like “SET NAMES utf8” as an initialization query, since the respective client libraries (libmysql or mysqlnd) will not be aware of this changed character set. For the usual connection encodings like “latin1” or “utf8”, this is not a problem, but in combination with multi-byte character sets that use bytes without the most significant bit set in multi-byte character sequences (such as GBK or Big5), this may result in incorrectly quoted strings, which could ultimately result in vulnerabilities in applications. For AgaviMysqlDatabase and AgaviMysqliDatabase, the simple fix is to remove the “SET NAMES” query from “init_queries” and use the new option “charset” instead. For AgaviPdoDatabase, a similar approach (specifying the charset in the DSN) works for PHP version 5.3.6 and newer, but in older versions of PHP, the PDO MySQL driver ignores the “charset” option in the DSN. If your my.cnf configuration file specifies “latin1” and you simply switch to UTF-8 using “SET NAMES utf8”, this is no problem, but for some charsets, this may result in the quoting issues mentioned above. This is why AgaviPdoDatabase requires you to set “warn_mysql_charset” to false in such environments. With this flag explicitly disabled, it will also be possible to use “SET NAMES” init queries again, but be advised that this is at your own risk. You may mitigate the risks by using native prepared statements (disable PDO::ATTR_EMULATE_PREPARES), but this will not affect calls to PDO::quote() or prepared statements where this flag has been overridden.

Passing an empty string as the scheme in the options array when generating a URL with AgaviWebRouting will now produce a protocol-relative URL starting with “//”. This is useful for generated content embedded by third party sites that use both normal and secured HTTP transports.

AgaviMysqlSessionStorage and AgaviPdoSessionStorage have improved update and insert behaviors for MySQL that should eliminate occasional non-critical “exception thrown without a stack frame” errors on shutdown when no session data (including the timestamp) was modified.

AgaviCreoleDatabase, AgaviCreoleSessionStorage and support for Propel versions older than 1.3 in AgaviPropelDatabase have been marked deprecated and are slated for removal in Agavi 1.1.

The recommended URL rewrites in Apache .htaccess files have been adjusted once more to disable the DirectorySlash directive.

X-Forwarded-Proto style HTTPS indicators as used on Amazon’s Elastic Load Balancers are now supported for the “HTTPS” data source key in AgaviWebRequest.

The timezone database has been updated to 2011g.

As usual, the CHANGELOG has a comprehensive list of all changes. A final release will follow shortly if no show-stoppers are found.

Agavi 1.0.4 released!

Sun, 19 Dec 2010 19:23:00 +0000

The final version of Agavi 1.0.4 is now available, our biggest maintenance release yet!

You can download it at agavi.org or install it via the pear.agavi.org PEAR channel.

Here is a summary of the most important changes over 1.0.3 (for a full list, please refer to the RELEASE_NOTES and CHANGELOG files):

Support for the Windows Azure platform
AgaviXsltRenderer
AgaviDoctrineDatabase lazy initialization
Logging enhancements
AgaviBooleanValidator and other validation improvements and fixes
Updated timezone database and Schematron implementation
PHPUnit is no longer bundled with Agavi
Resolves compatibility problems with Phing 2.4.4 due to a BC break
Improved exception rendering
New rewrites in the default .htaccess files

The ability to extract subpatterns in AgaviRegexValidator introduced in 1.0.4 RC1 has been modified, please see the associated ticket for details.

The next release will be 1.0.5 with a bunch of enhancements as detailed earlier this week; an RC should be out before the end of this year.

Happy holidays to you and your families!

Agavi 1.0.4 RC1 released!

Tue, 14 Dec 2010 18:33:00 +0000

It’s been quite a while since our last release, so we’re happy to announce the availability of Agavi 1.0.4 RC1, our biggest maintenance release yet!

You can download it at agavi.org or install it via the pear.agavi.org PEAR channel.

Here is a summary of the most important changes in this release (for a full list, please refer to the RELEASE_NOTES and CHANGELOG files):

Support for the Windows Azure platform
AgaviXsltRenderer
AgaviDoctrineDatabase lazy initialization
Logging enhancements
AgaviBooleanValidator and other validation improvements and fixes
Updated timezone database and Schematron implementation
PHPUnit is no longer bundled with Agavi
Resolves compatibility problems with Phing 2.4.4 due to a BC break
Improved exception rendering

Please test this version thoroughly and report any issues. If there aren’t any major roadblocks, a final release should follow in a few days.

Version 1.0.5 is also being planned already and will feature a few enhancements such as support for the Doctrine 2, Smarty 3 and Twig libraries as well as more lenient parsing of forms by FPF in both HTML and XHTML mode.

Agavi 1.0.3 released!

Fri, 16 Apr 2010 18:05:00 +0100

Agavi 1.0.3 has just been released and can be downloaded at agavi.org.

Only very few changes were done since RC1, so I’ll once more leave it to the RELEASE_NOTES to list the most important changes:

This maintenance release fixes a number of issues over Agavi 1.0.2 and features a few minor enhancements.

Several issues and inconsistencies have been fixed in AgaviNumberValidator and AgaviDecimalFormatter::parse(). Parsing behavior is now consistent with the NumberFormatter implementations of ICU and ext/intl. No integer overflow occurs when checking against min and max values, and the validator accepts non-strings as input. Exporting of the validated value is now possible, in which case the original argument is not mutated. Please note that the “number” validator shortcut definition specifies “int” as the default value for the “type” parameter. No value for “type” will automatically lead to integer or float values, depending on the size of the number in the input string. A cast to the specified type is now automatically performed if no “cast_to” parameter is given.

All file validators now perform case-insensitive checks against the list of allowed file extensions, which may now also be a parameter array instead of a space-separated list of values.

This release bumps the minimum required version of Phing to 2.4.0; this was necessary to fix some bugs with the build system that were next to impossible to fix for both 2.3 and 2.4 (which drastically changed some internals structures) at the same time.

An issue relating to the behavior of DOM in PHP 5.1.12 and PHP 5.2.3 has been addressed. Running Agavi 1.0.3 might be necessary under certain circumstances when using these versions of PHP.

A couple of minor fixes have been made to the routing; most notably, overwriting of routes in other <ae:configuration> blocks now always works as expected, even when using implied routes.

Some default code templates have been updated; most notably, caching configuration files now explicitly disable caching, as otherwise, Actions that do not use a View would be cached automatically.

The timezone database has been updated to version 2010h.

For a full list of changes, please refer to the CHANGELOG.

Agavi on the Azure Platform

Tue, 13 Apr 2010 20:52:53 +0100

The next release of Agavi will have initial support for running applications on the Microsoft Windows Azure platform, as well as a database adapter for the new ext/sqlsrv driver to communicate with Microsoft SQL Server and support for the IIS7 web server, which now finally has a very nice rewrite module.

As always with these kind of features, the biggest task was getting to know the platform and its components. I spent a good amount of time setting up the necessary tools, learning the finer details of Azure deployment, researching on how the load balancing worked, finding the right rewrite rules for IIS and bending ext/sqlsrv to my will.

In Agavi itself, you will merely find a new session storage class (for cloud-based sessions), a new database adapter class to talk to MSSQL, and some minor modifications to the web request and routing implementations to deal with IIS7’s URL decoding behavior. The latter is still a work in progress, and requires a lot of testing since unfortunately, Apache/IIS/lighttpd/… all have vastly different behavior especially when it comes to URL decoding.

All of these updates will be in Agavi 1.0.4; I plan on releasing 1.0.3 as a final first (it’s been out there long enough in RC state now).

Expect a blog post with examples on how to get your Agavi app up and running on Azure very soon; especially with the Toolkit for Eclipse, it’s a remarkably convenient way of testing and deploying your applications on a cloud infrastructure. And there will, of course, be another article with a wrap-up of last week’s Jump In Camp.

Jump In! Camp

Wed, 07 Apr 2010 09:42:21 +0100

Jump In! Developers’ Camp has started today at a beautiful resort near Zürich, Switzerland. Over the next three days, we’ll have workshops focusing on the topics of cloud computing, the Microsoft Azure platform. There are quite a bunch of fellow Open Source developers here, and it’ll be exciting to sit together and hack away on various features.

For Agavi, this means that work will be done this week on native support for the Azure cloud computing platform, support for IIS7 and the new SQL Server Driver for PHP. I’m also hoping to get some support in for the table and queue stores the Azure platform offers, although that will require a lot work on the new storage subsystem planned for Agavi 1.1 that will be difficult to squeeze into the next three days.

In addition to that, I’ve been working on support for both input types (finally!) and a new security infrastructure that decouples authentication and authorization and will allow you, among other things, to access validated request data, so you can for instance implement an ACL subsystem to perform authorization checks directly on your business objects.

I’ll blog several updates over the next few days, so stay tuned!

– David

Agavi 1.0.3 RC1 released!

Mon, 22 Mar 2010 22:23:24 +0000

Agavi 1.0.3 RC1 is ready for testing and can be downloaded at www.agavi.org.

I’ll quote right from the release notes to give you an overview of the most important changes. Please help us test this release thoroughly!

This maintenance release fixes a number of issues over Agavi 1.0.2 and features a few minor enhancements.

Several issues and inconsistencies have been fixed in AgaviNumberValidator and AgaviDecimalFormatter::parse(). Parsing behavior is now consistent with the NumberFormatter implementations of ICU and ext/intl. No integer overflow occurs when checking against min and max values, and the validator accepts non-strings as input. Exporting of the validated value is now possible, in which case the original argument is not mutated. Please note that the “number” validator shortcut definition specifies “int” as the default value for the “type” parameter. No value for “type” will automatically lead to integer or float values, depending on the size of the number in the input string. A cast to the specified type is now automatically performed if no “cast_to” parameter is given.

All file validators now perform case-insensitive checks against the list of allowed file extensions, which may now also be a parameter array instead of a space-separated list of values.

This release bumps the minimum required version of Phing to 2.4.0; this was necessary to fix some bugs with the build system that were next to impossible to fix for both 2.3 and 2.4 (which drastically changed some internals structures) at the same time.

An issue relating to the behavior of DOM in PHP 5.1.12 and PHP 5.2.3 has been addressed. Running Agavi 1.0.3 might be necessary under certain circumstances when using these versions of PHP.

A couple of minor fixes have been made to the routing; most notably, overwriting of routes in other blocks now always works as expected, even when using implied routes.

Some default code templates have been updated; most notably, caching configuration files now explicitly disable caching, as otherwise, Actions that do not use a View would be cached automatically.

The timezone database has been updated to version 2010f.

For a full list of changes, please refer to the CHANGELOG.

Agavi at ConFoo

Tue, 09 Mar 2010 20:33:00 +0000

The beacon of hope that is Agavi is carried to Montréal this week, where the ConFoo Web Tech conference takes place.

David will be giving a talk on Agavi, and although we know that some of you will be there to watch it live, all those who couldn’t make it to Montréal (it’s a lovely city, so consider attending next year) need not despair, as at least the slides, and possibly also a recording of the talk will be posted after the conference.

If you’re there in person, make sure to catch a seat in one of the front rows so you can be first in line after the talk to grab one of the Agavi t-shirts that made the long journey to Canada.