10 essential MySQL tools for admins

By Daniel Nichter
Created 2011-08-01 03:00AM

MySQL is a complex system that requires many tools to repair, diagnose, and optimize it. Fortunately for admins, MySQL has attracted a vibrant community of developers who are putting out high-quality open source tools to help with the complexity, performance, and health of MySQL systems, most of which are available for free.

The following 10 open source tools are valuable resources for anyone using MySQL, from a stand-alone instance to a multiple-node environment. The list has been compiled with variety in mind. You will find tools to help back up MySQL data, increase performance, guard against data drift, and log pertinent troubleshooting data when problems arise.

There are several reasons why you should consider these tools instead of creating your own in-house tools. First, thanks to their wide use, they’re mature and field-tested. Second, because they are free and open source, they benefit from the knowledge and experience of the continually expanding MySQL community. Finally, these tools are actively developed, and many are professionally supported (either for free or commercially), so they continue to improve and adapt with the evolving MySQL industry.

Keep in mind that there are many more tools worthy of your attention. I have chosen to emphasize free and open source, and to err on the side of usefulness and usability. Also note that all are Unix command-line programs but one, in large part because MySQL is more widely deployed and developed on Unix systems.

Now, let’s meet the first of the 10 essential MySQL tools.

Essential MySQL tool No. 1: mk-query-digest
Nothing frustrates like slow MySQL performance. All too often, faster hardware is thrown at the problem, a solution that works only if hardware is in fact to blame. More often than not, poor performance can be attributed to slowly executing queries that are blocking other queries, creating a ripple effect of slow response times. Since it’s a lot cheaper to optimize queries than to upgrade hardware, the logical first step in MySQL optimization is query log analysis.

Database administrators should analyze query logs frequently, depending on the volatility of the environment. And if you’ve never performed query log analysis, it’s time to start, even if you are relying on third-party software, which is often assumed to be optimized when, in fact, it is not.

Today’s best query log analyzer is mk-query-digest. Co-written by Baron Schwartz and myself, it is actively developed, fully documented, and thoroughly tested. MySQL distributions include the query log analyzer mysqldumpslow, but the tool is outdated, poorly documented, and untested. Other query log analyzers, likemysqlsla, which I wrote several years ago, suffer the same problems as mysqldumpslow.

mk-query-digest analyzes query logs and generates reports with aggregated, statistical information about execution times and other metrics. Since query logs usually contain thousands, if not millions, of queries, query log analysis requires a tool.

mk-query-digest can help you find the queries that take the longest time to execute as compared to other queries. Optimizing these slow queries will make MySQL run faster by reducing the greatest delays. The real art of query optimization is more nuanced, but the basic goal is the same: find slow queries, optimize them, and increase query response times.

The tool is easy to use; executing mk-query-digest slow-query.log will print the slowest queries in slow-query.log. The tool includes support for “query reviews,” for reporting queries you have not yet seen or approved, making frequent log analyses quick and efficient.

Download: http://maatkit.org/get/mk-query-digest
Maintainers: Daniel Nichter and Baron Schwartz
More info: http://maatkit.org/ | http://code.google.com/p/maatkit/

Essential MySQL tool No. 2: mydumper
Being able to generate data dumps quickly is vital for backups and server cloning. Unfortunately, mysqldump, which ships with MySQL distributions, is single-threaded and, thus, too slow for data-intensive jobs. Thankfully, the modern replacement, mydumper, uses multiple threads, making it 10 times as faster than mysqldump.

Also known as MySQL Data Dumper, this tool does not manage backup sets, differentials, or other parts of a complete backup plan. It just dumps data from MySQL as quickly as possible, enabling you to complete backups under tight time constraints, such as overnight, while employees are offline, or to perform backups more frequently than you would with mysqldump.

One technical point to know about mydumper is that it locks tables, so it is not the ideal tool for performing backups during operating hours. Then again, professional data recovery costs hundreds of dollars per hour, and you always get a bill even if the data isn’t recoverable. mydumper is free and well worth exploring for even basic backups.

mydumper also comes in handy when cloning servers. Other tools perform complete hard drive duplications, but when all you need is MySQL data, mydumper is the fastest way to get it. Servers provisioned in a cloud are particularly suited to cloning using mydumper. Just dump your MySQL data from an existing server and copy it to the new instance.

Cloning is worthwhile for creating slave servers, benchmarking, and profiling, but nowhere is it more vital than in testing and development. Being able to spin up a replica for quick testing before going live is essential for dynamic MySQL environments. With mydumper, you can quickly create a server that is nearly identical to your production server, enabling your test results to better mimic production results.

Download: https://launchpad.net/mydumper/+download
Maintainers: Domas Mituzas, Andrew Hutchings, Mark Leith
More info: http://www.mydumper.org/ | https://launchpad.net/mydumper/

Essential MySQL tool No. 3: xtrabackup and xtrabackup-manager
If your databases are in use every day, all day, giving you no “overnight” during which tables can be locked for backup, xtrabackup is your solution. Also known as Percona XtraBackup, this tool performs nonblocking backups and is the only free, open source tool that can do this. By comparison, proprietary nonblocking backup software can cost more than $5,000 per server.

xtrabackup also offers incremental backups, allowing you to back up only the data that has changed since the last full backup. Adding incremental backups to your backup process is powerful, given the reduced performance hit of these tremendously smaller backups.

Furthermore, another project has grown up around xtrabackup that makes managing a full backup plan even easier: xtrabackup-manager. Although this tool is new and still in development, it holds a lot of potential because it offers advanced features like rotating backups with groups and backup set expiring. Together, xtrabackupand xtrabackup-manager are a formidable and free backup solution.

Download: http://www.percona.com/software/percona-xtrabackup/downloads/
Maintainer: Percona
More info: http://www.percona.com/docs/wiki/percona-xtrabackup:start | https://launchpad.net/percona-xtrabackup

Download: http://code.google.com/p/xtrabackup-manager/
Maintainer: Lachlan Mulcahy
More info: http://code.google.com/p/xtrabackup-manager/ | http://mysqlsoapbox.blogspot.com/

Essential MySQL tool No. 4: tcprstat
tcprstat is probably the most esoteric of the 10 on this list. The tool monitors TCP requests and prints statistics about low-level response times. When you become familiar with the response time way of thinking about performance, the payoff of tcprstat is significant.

The principle is elaborated in the book “Optimizing Oracle Performance” by Cary Millsap and Jeff Holt, and it applies equally well to MySQL. The basic idea is that a service, in this case MySQL, accepts a request (query), fulfills that request (execution time), and responds with results (result set). The service’s response time is the time span between receiving a request and sending a response. The shorter the response time, the more requests can be served in the same amount of time.

Parallel processing and other low-level factors play a significant part here, but the simplified upshot is that there are 28,800 seconds in an 8-hour workday, so reducing response times by just four-tenths of a second (from 0.5 to 0.1 second) results in 230,400 more requests served each day. tcprstat helps you achieve this.

I have only enough space in this article to pique your curiosity, so I’ll finish this tool’s introduction by telling you the first step toward getting started with MySQL response time optimization: read “Optimizing Oracle Performance.” Then start using tcprstat.

Download: (source) https://launchpad.net/tcprstat | (binary) http://www.percona.com/docs/wiki/tcprstat:start
Maintainers: Percona
More info: http://www.percona.com/docs/wiki/tcprstat:start | https://launchpad.net/tcprstat

Essential MySQL tool No. 5: mk-table-checksum
“Data drift” is a significant problem for dynamic MySQL environments. This problem, wherein slave data becomes out of sync with the master, is often caused by writing data to a slave or executing certain nondeterministic queries on the master. What’s worse is that the data differences may go unnoticed until they become crippling. Entermk-table-checksum, a tool that performs the complex, sensitive calculations necessary to verify the data in two or more tables is identical.

mk-table-checksum works with both stand-alone servers and servers in a replication hierarchy, where the tool’s greatest value is easily seen. Verifying table data between a master and a slave must account for replication consistency. Because changes to the master are replicating to slaves with some amount of time delay (“lag”), simply reading data from the servers is an unreliable way to verify consistency, given that the data is constantly changing and incomplete until fully replicated. Locking tables and waiting for all data to replicate would allow consistent reads, but to do so would mean effectively halting the servers. mk-table-checksum allows you to perform nonblocking, consistent checksums of master and slave data. (For technical details on how this is accomplished, see the tool’s documentation.)

Apart from replication consistency, there are other problems with verifying data. Table size is one of them. The MySQL command CHECKSUM TABLE is sufficient for small tables, but large tables require “chunking” to avoid long locks or overloading CPU or memory resources with checksum calculations.

Chunking solves a second problem: the need for regular data-consistency checks. While data drift can be a one-time occurrence, often it is recurring. mk-table-checksum is designed to continuously check tables, vetting certain chunks one run and other chunks the next run until eventually the whole table has been checked. The ongoing nature of this process helps ensure that recurring drift is corrected.

Download: http://maatkit.org/get/mk-table-checksum
Maintainers: Daniel Nichter & Baron Schwartz
More info: http://maatkit.org/ | http://code.google.com/p/maatkit/

Essential MySQL tool No. 6: stalk and collect
Problems have a way of waiting until you’re not looking or at home sleeping to occur, and diagnosing them after the fact is sometimes impossible without data about the state of MySQL and the server at the time of the problem. The natural inclination is to write your own script to wait for or detect a problem and then start logging extra data because, after all, no one knows your system better than you. The problem is, you know your system when it’s working, and if you knew the kinds of problems the system would have, you would simply fix them rather than try to capture and analyze them.

Thankfully, those who specialize in knowing when MySQL is not working, and in fixing the problems, have written a duo of tools called stalk and collect. The first tool waits for certain conditions to become true before running an instance of the second tool. That seems trivial, but these tools are made efficient by certain details addressed.

Firstly, stalk runs collect in configurable intervals, keeping you from logging too much redundant data, which can obfuscate postproblem analysis. Secondly,collect gathers not only the standard information that MySQL can report about itself but a lot more data that you might not have thought to include: lsof, strace, tcpdump, and so on. Thus, if you end up having to consult with a professional who specializes in fixing MySQL problems, you will have all the data that they need.

stalk and collect are configurable, so they can be used for almost any problem. The one requirement is a definable condition to establish a trigger for stalk. If multiple conditions signal the problem, then you may also need to consult with a professional for a more extensive review of your MySQL environment because problems can appear in MySQL even though the underlying cause is elsewhere.

stalk and collect can be used proactively, too. For example, if you know that there should never be more than 50 active MySQL connections at a time, then you could proactively monitor this stalk, making these tools helpful both for problems that you know and problems that you have not yet seen.

Download: http://aspersa.googlecode.com/svn/trunk/stalk | http://aspersa.googlecode.com/svn/trunk/collect
Maintainer: Baron Schwartz
More info: http://aspersa.googlecode.com/svn/html/index.html | http://code.google.com/p/aspersa/

Essential MySQL tool No. 7: mycheckpoint
You don’t always want to wait for something to go wrong before addressing a problem, and dashboards provide an essential way for you to monitor your MySQL environment for potential problems before they arise.

There are many free and commercial monitoring applications for MySQL, some MySQL-specific and others generic with MySQL plug-ins or templates. mycheckpointis notable because it is free, open source, MySQL-specific, and full-featured.

Like many monitoring solutions, mycheckpoint is Web-based. Here’s an example chart:

mycheckpoint can be configured to monitor both MySQL and server metrics, like InnoDB buffer pool flushes, temporary tables created, operating system load, memory usage, and more. If you don’t like charts, mycheckpoint can also generate human-readable reports.

As with stalk, alert conditions can be defined with email notifications, but no secondary tool like collect will be run to log additional troubleshooting data. Another useful feature is mycheckpoint’s ability to monitor MySQL variables to detect changes that can lead to problems, or signal that someone has modified MySQL when they shouldn’t have.

Monitoring MySQL isn’t just for data centers or large deployments. Even if you have a single MySQL server, monitoring is essential; as with your vehicle, there’s a lot to know about the system while it’s running to help you foresee or avoid malfunctions. mycheckpoint is one solution among many worth trying.

Download: http://code.google.com/p/mycheckpoint/downloads/list
Maintainer: Shlomi Noach
More info: http://code.openark.org/forge/mycheckpoint

Essential MySQL tool No. 8: shard-query
Queries against partitioned or sharded data sets can be accelerated dramatically using shard-query, which parallelizes certain queries behind the scenes. Queries that use the following constructs can benefit from shard-query’s parallel execution:

• Subqueries in the FROM clause
• UNION and UNION ALL
• IN
• BETWEEN

Aggregate functions SUM, COUNT, MIN, and MAX can be used with those constructs, too. For example, this query can be executed in parallel by shard-query:

SELECT DayOfWeek, COUNT(*) AS c
FROM ontime_fact
JOIN dim_date USING(date_id)
WHERE Year
BETWEEN 2000 AND 2008
GROUP BY DayOfWeek
ORDER BY c DESC;

Benchmarks show that parallelizing that query reduces its response time by roughly 85 percent, from 21 seconds to 3.

shard-query is not a stand-alone tool; it requires other programs like Gearman, and it’s relatively complex to set up. But if your data is partitioned and your queries use any of the constructs listed above, then the benefits are worth the effort.

Download: (svn checkout) http://code.google.com/p/shard-query/source/checkout
Maintainer: Justin Swanhart
More info: http://code.google.com/p/shard-query/

Essential MySQL tool No. 9: mk-archiver
As tables become larger, queries against them can become slower. Many factors influence response times, but if you have optimized everything else and the only remaining suspect is a very large table, then archiving rows from that table can restore fast query-response times.

Unless the table is unimportant, you should not brazenly delete rows. Archiving requires finesse to ensure that data is not lost, that the table isn’t excessively locked, and that the archiving process does not overload MySQL or the sever. The goal is an archiving process that is reliable and unnoticeable except for the beneficial effect of reducing query times. mk-archiver achieves all this.

mk-archiver has two fundamental requirements, the first of which is that archivable rows must be identifiable. For example, if the table has a date column and you know that only the last N years of data are needed, then rows with dates older than N years ago can be archived. Moreover, a unique index must exist to help mk-archiver identify archivable rows without scanning the entire table. Scanning a large table is costly, so an index and specific SELECT statements are used to avoid table scans.

In practice, mk-archiver automatically handles the technical details. All you have to do is tell it what table to archive, how to identify archivable rows, and where to archive those rows. These rows can be purged, copied to another table, or written to a dump file for future restoration if needed. Once you’re comfortable with the tool, there are many options to fine-tune the archiving process. Also, mk-archiver is pluggable, so it can be used to solve complex archiving needs without patching the code.

Download: http://maatkit.org/get/mk-archiver
Maintainers: Daniel Nichter and Baron Schwartz
More info: http://maatkit.org/ | http://code.google.com/p/maatkit/

Essential MySQL tool No. 10: oak-security-audit
When was the last time you audited the security of your MySQL servers? You’re not alone if “never” is the answer. There are many companies that provide security audits, but unless nothing ever changes after those audits, then the security of your MySQL environment should be checked regularly.

External threats are one obvious reason to enforce MySQL security, but internal threats like current or former employees are often more dangerous because they are (or were) trusted. Security is also important for enforcing privacy (medical/HIPAA regulations), preventing accidental access (for example, logging into the production server instead of the development server), or enabling third-party programs to interact with your systems.

For those looking to increase the security of their deployments, oak-security-audit is a worthwhile, free, open source tool that performs basic MySQL security audits. It doesn’t require any setup; just run it against your MySQL servers, and it prints a report with risks and recommendations about accounts, account privileges, passwords, and some general best practices, like disabling network access. Here’s a snippet of a report:

– Looking for anonymous user accounts
– ———————————–
– Passed
–
– Looking for accounts accessible from any host
– ———————————————
– Found 1 accounts accessible from any host. Recommended actions:
RENAME USER ‘msandbox’@'%’ TO ‘msandbox’@'<specific host>’;

oak-security-audit focuses just on MySQL security, so it’s not a replacement for a full system security audit by a human, but it’s a great first line of defense that is easy to use. You could run it weekly with cron and have the reports emailed to you.

Download: http://openarkkit.googlecode.com/svn/trunk/openarkkit/src/oak/oak-security-audit.py
Maintainer: Shlomi Noach
More info: http://openarkkit.googlecode.com/svn/trunk/openarkkit/doc/html/oak-security-audit.html

The following snippet does exactly that. Obviously, it doesn’t have a lot of functionality, it just outputs the current user’s data if the user is authenticated.  Note that in this example the code is located in a subdirectory of WordPress’ root directory (hence, the ../ in include ‘../wp-config.php’), but it should work from anywhere, as long as wp-config.php is reachable for inclusion.

include '../wp-config.php';
wp_get_current_user();
if ($current_user->ID == 0) {
  $location = 'Location: ' . site_url('')
    . '/wp-login.php?redirect_to='
    . urlencode($_SERVER['PHP_SELF']);
  header($location);
  echo "<p>You are not logged in.</p>\r\n";
} else {
  echo "<h2>WP: wp_get_current_user()</h2>\r\n";
  echo "<pre>";
  var_dump($current_user);
  echo "</pre>\r\n";
}

Note the picture of camel.  I wonder if this is a joke by the server’s operators or a friendly prank by someone in the Perl community…

From New York Times:

A Quantum Theory of Mitt Romney

By DAVID JAVERBAUM

THE recent remark by Mitt Romney’s senior adviser Eric Fehrnstrom that upon clinching the Republican nomination Mr. Romney could change his political views “like an Etch A Sketch” has already become notorious. The comment seemed all too apt, an apparent admission by a campaign insider of two widely held suspicions about Mitt Romney: that he is a) utterly devoid of any ideological convictions and b) filled with aluminum powder.

The imagery may have been unfortunate, but Mr. Fehrnstrom’s impulse to analogize is understandable. Metaphors like these, inexact as they are, are the only way the layman can begin to grasp the strange phantom world that underpins the very fabric of not only the Romney campaign but also of Mitt Romney in general. For we have entered the age of quantum politics; and Mitt Romney is the first quantum politician.

A bit of context. Before Mitt Romney, those seeking the presidency operated under the laws of so-called classical politics, laws still followed by traditional campaigners like Newt Gingrich. Under these Newtonian principles, a candidate’s position on an issue tends to stay at rest until an outside force — the Tea Party, say, or a six-figure credit line at Tiffany — compels him to alter his stance, at a speed commensurate with the size of the force (usually large) and in inverse proportion to the depth of his beliefs (invariably negligible). This alteration, framed as a positive by the candidate, then provokes an equal but opposite reaction among his rivals.

But the Romney candidacy represents literally a quantum leap forward. It is governed by rules that are bizarre and appear to go against everyday experience and common sense. To be honest, even people like Mr. Fehrnstrom who are experts in Mitt Romney’s reality, or “Romneality,” seem bewildered by its implications; and any person who tells you he or she truly “understands” Mitt Romney is either lying or a corporation.

Nevertheless, close and repeated study of his campaign in real-world situations has yielded a standard model that has proved eerily accurate in predicting Mitt Romney’s behavior in debate after debate, speech after speech, awkward look-at-me-I’m-a-regular-guy moment after awkward look-at-me-I’m-a-regular-guy moment, and every other event in his face-time continuum.

The basic concepts behind this model are:

Complementarity. In much the same way that light is both a particle and a wave, Mitt Romney is both a moderate and a conservative, depending on the situation (Fig. 1). It is not that he is one or the other; it is not that he is one and then the other. He is both at the same time.

Probability. Mitt Romney’s political viewpoints can be expressed only in terms of likelihood, not certainty. While some views are obviously far less likely than others,no view can be thought of as absolutely impossible. Thus, for instance, there is at any given moment a nonzero chance that Mitt Romney supports child slavery.

Uncertainty. Frustrating as it may be, the rules of quantum campaigning dictate that no human being can ever simultaneously know both what Mitt Romney’s current position is and where that position will be at some future date. This is known as the “principle uncertainty principle.”

Entanglement. It doesn’t matter whether it’s a proton, neutron or Mormon: the act of observing cannot be separated from the outcome of the observation. By asking Mitt Romney how he feels about an issue, you unavoidably affect how he feels about it. More precisely, Mitt Romney will feel every possible way about an issue until the moment he is asked about it, at which point the many feelings decohere into the single answer most likely to please the asker.

Noncausality. The Romney campaign often violates, and even reverses, the law of cause and effect. For example, ordinarily the cause of getting the most votes leads to the effect of being considered the most electable candidate. But in the case of Mitt Romney, the cause of being considered the most electable candidate actually produces the effect of getting the most votes.

Duality. Many conservatives believe the existence of Mitt Romney allows for the possibility of the spontaneous creation of an “anti-Romney” (Fig. 2) that leaps into existence and annihilates Mitt Romney. (However, the science behind this is somewhat suspect, as it is financed by Rick Santorum, for whom science itself is suspect.)

What does all this bode for the general election? By this point it won’t surprise you to learn the answer is, “We don’t know.” Because according to the latest theories, the “Mitt Romney” who seems poised to be the Republican nominee is but one of countless Mitt Romneys, each occupying his own cosmos, each supporting a different platform, each being compared to a different beloved children’s toy but all of them equally real, all of them equally valid and all of them running for president at the same time, in their own alternative Romnealities, somewhere in the vast Romniverse.

And all of them losing to Barack Obama.

David Javerbaum is the author of “The Last Testament: A Memoir by God.”

http://maps.stamen.com

Played around with it, and it was very robust while showing San Francisco, but slowed down considerably when I tried Los Angeles.  Could it be that SF maps are cached server-side as SF is the default location displayed at the developer’s site?  Or could have been a one-time fluke after all…

Verizon security report:
Hacktivism up, insider threats down

By InfoWorld Tech Watch
Created 2012-03-22 03:00AM

Over 855 data breaches resulting in the compromise of more than 174 million records around the world provided rich source material for the 2012 Verizon Data Breach Investigations Report, a highly respected analysis that includes data from the U.S. Secret Service, four other investigative agencies, and the telecom giant’s own vast trove of information.

The biggest change, unsurprisingly, was the 58 percent rise in stolen records attributed to hacktivism. The Anonymous hacker group surely accounted for a sizable percentage of this activity, although the report doesn’t break that out. Clearly it has become socially acceptable in some circles to publicly announce a high-profile target and recruit a mob of volunteers.

Hacktivism probably contributed to the report’s finding that 98 percent of all attacks were committed by outsiders. Nearly 70 percent of breaches originated in Eastern Europe, which dispels the notion of Chinese dominance (at least this year). Less than 25 percent of attacks originated from the United States. Anecdotally, even some of the biggest U.S. cyber arrests included Eastern European players located in the United States.

The best news in the report is that the percentage of internal attackers has diminished significantly over the last few years, making up only 4 percent of last year’s attacks, down from a high of 48 percent in 2009. Less than 1 percent of breaches resulted from hacks by business partners.

Accounting for 10 percent of all breached records, physical attacks continue to be a factor, from card skimming to stolen storage devices and other physical tampering. Although card skimming was down slightly, skimming combined with ATM PIN camera recording accounted for 35 percent of physical attacks, which is more than double the incidents from the previous year.

The elephant in the room remains the same: The report estimates that 97 percent of data breaches could have been avoided if simple security measures were in place. Worse, 96 percent of victims that were required to comply with PCI DSS guidelines — security measures stipulated by the credit card industry — were not compliant at the time of the breach.

In almost every case, if the victims had applied normal countermeasures the data breaches probably wouldn’t have occurred. No victims are named in the report, but two of the biggest breaches in 2011 — involving Sony and RSA and resulting in the exposure of tens of millions of records — were due to unpatched software. The report again affirms that specialized antihacking tools and software are not needed. What’s needed is more consistent application of the basics.

Worse, 85 percent of victims were unaware of the compromise for weeks to months. And when they discovered the breach, 92 percent learned about it from a third party. You can be sure that event log monitoring vendors will include Verizon’s data in their ad campaigns.

A must-read for most C-level officers, the Verizon Data Breach Investigations Report has provided an invaluable public service for five years. Yet, by and large, the underlying vulnerabilities have stayed the same. Do something about them, and don’t let your company become next year’s statistic.

By Edward Robinson - Mar 13, 2012
Bloomberg Markets Magazine

Ken Major climbs the steps of a county courthouse in a San Francisco suburb with $500,000 in cashier’s checks in one hand and a list of addresses in the other. Major is a buyer for Waypoint Real Estate Group LLC, an Oakland-based investment firm that’s scooping up foreclosed homes in California.

On this December afternoon, he joins a dozen house flippers as an auctioneer starts hawking the latest batch of defaulted properties to hit the market. Major bids on a three-bedroom house in Antioch, and after other buyers counter, he wins at $147,600.

“We got it,” he mutters into a mobile-phone mic dangling from his ear. The house was valued at more than $400,000 in 2006, Bloomberg Markets magazine reports in its April issue.

Waypoint, a private-equity real-estate fund with $150 million in assets, is pioneering a new approach to making money from the housing crash. Since 2007, investors have been trolling the cratered suburbs stretching from California to Florida (SPCSMIA) for cheap houses to flip. And firms such as PennyMac Mortgage Investment Trust have sought value in subprime-mortgage-backed securities.

Waypoint, which owns 1,100 houses and is buying five more a day, is betting that converting foreclosures into rentals is a better way to make a profit. Other firms, such as Landsmith LP in San Francisco, are now cropping up and pursuing the same strategy in Arizona, California andNevada.

‘Yields Are Awesome’

With many suburban homes selling for half their peak values and demand for rentals from prospective tenants climbing, Waypoint was earning an 8 to 9 percent return on its capital as of Dec. 31, according to a quarterly report it sends to clients. That beats the 6.3 percent gain in the BI NA Multifamily REIT (BRFREITC) Index, which tracks the performance of 27 apartment building operators.

The cost of renting in the U.S. reached an all-time high compared with that of buying a home at the end of last year, indicating it’s a good time for investors to purchase, Deutsche Bank AG (DBK) analysts said in a note today. Should property values rebound, Waypoint may earn at least 20 percent from appreciation in an eventual sale of the houses, says Colin Wiel, who co- founded the firm in 2008 after backing technology startups as an angel investor.

“I never thought I’d be rolling up single-family homes,” Wiel says. “But the yields are awesome.”

Wiel and Waypoint co-founder Doug Brien make an unlikely pair of real-estate entrepreneurs. Wiel, 45, is a mechanical engineer who designed braking systems for jetliners at Boeing Co. (BA) in the 1990s. And Brien, 41, is a former placekicker who won a Super Bowl with the San Francisco 49ers in 1995 before earning a postgraduate degree in business at Tulane University in New Orleans.

$3 Trillion Market

In starting Waypoint, Wiel and Brien set out to show institutional investors that by using technology they could amass single-family homes the same way Sam Zell’s Equity Group Investments Inc. (EQR) and other real-estate giants gather apartment units in cities from New York to San Francisco.

The home rental market boasts a total property value of $3 trillion, according to Morgan Stanley (MS) housing analyst Oliver Chang. Yet institutions have long shunned it as too scattered and impractical to be profitable.

Wiel and Brien are using cloud computing, proprietary algorithms and iPads to create a virtual assembly line for buying, renovating and renting houses on a large scale. They’re also betting that many former homeowners who have jobs but couldn’t afford their mortgages will still want to live in the same communities as renters.

Acquire And Convert

“The economics never made sense for a big investor to come into the market, and the technology for managing all that complexity didn’t previously exist,” says Brien, who still possesses the steely stare of a field goal kicker. “The confluence of those two events has provided a window of opportunity for large investors to enter this space.”

Last year, Columbia University’s $8 billion endowment invested $25 million with Waypoint. In January, GI Partners, a Menlo Park, California-based private-equity firm that manages money for the California Public Employees’ Retirement System and other pension plans, agreed to invest up to $400 million with Waypoint and acquire a minority stake in the firm.

The same month, Oaktree Capital Management LP, the Los Angeles investment firm co-founded by billionaire Howard Marks, announced a $450 million deal with Santa Ana, California-based Carrington Capital Management LLC to acquire and convert foreclosed single-family homes into rental properties. Carrington already rents out more than 3,000 houses in California and other states.

Labor Intensive

Starwood Capital Group LLC is poised to enter the foreclosure-to-rental market, according to an investor familiar with its plans. So, too, is Zell and the real-estate arm of Apollo Investment Management LLC.

Spokespersons for Zell, Apollo and Starwood declined to comment.

“Until last year, single-family-home rentals was a mom and pop market,” says Stephen Duffy, an investment banker at Moss- Adams Capital LLC, an Irvine, California-based firm that finances real-estate investments. “Now, it’s grabbed the attention of institutional private equity because foreclosures haven’t cleared and these properties can generate high yields for years.”

Waypoint and its rivals may eventually spin off pools of single-family home rentals into real estate investment trusts. Still, even the best technology cannot replace the labor- intensive process of acquiring and leasing thousands of houses scattered across scores of zip codes.

Nascent Market

Waypoint’s researchers must plumb school desirability ratings, crime statistics and other hyperlocal data to ascertain the income value of each house. Its title agents must often disentangle foreclosures from second mortgages and liens. And leasing representatives have to find qualified renters in communities struggling with high unemployment rates.

“This is a nascent market, and the model still hasn’t proved out,” says Rick Magnuson, executive managing director of GI Partners, which has $6 billion under management. “But we believe this rental strategy will produce good economic returns for our investors and help arrest the housing market’s slide.”

The Obama administration is poised to tap the foreclosure- to-rental approach as officials struggle to turn around the housing market. The president’s push to have mortgage providers make loans more affordable for homeowners has done little to stem foreclosures, says Ginna Green, a spokeswoman for the Center for Responsible Lending, a Durham, North Carolina-based consumer advocacy organization.

Worst Is Over

There were almost 2 million U.S. foreclosures in 2011, down 31 percent from 2010. As many as 10 million borrowers may default over the next few years if the markets continue to deteriorate, says Laurie Goodman, an analyst at Amherst Securities Group LP in New York. Even thoughmortgage rates are hovering at a historic low of 3.8 percent, consumers bought only 324,000 new homes last year, the poorest annual performance since 1963.

On Feb. 9, the U.S. Department of Justice and 49 states agreed to end a probe into abusive mortgage practices at Bank of America Corp., JPMorgan Chase & Co. and three other banks after striking a $25 billion settlement with the companies. The landmark agreement, which will provide debt relief to homeowners, should help rescue many delinquent borrowers and buoy the confidence of would-be homebuyers and lenders that the worst is over, says Ivy Zelman, CEO of Zelman & Associates LLC?, a Cleveland-based research firm.

Repossessed Homes

Even so, the settlement may not be large enough to reboot a housing market saddled with $700 billion in underwater mortgages. She says institutional investors eyeing the rental market have the capital to absorb thousands of dwellings and slow the market’s decline.

Investors are already having an effect: The supplies of homes for sale in Phoenix (SPCSPHXS), Orlando, Florida and other hard-hit markets have fallen more than 60 percent from their post-crash highs as bargain hunters scoop up foreclosures.

“Investors are aggressive about buying these homes in front of the government program,” Zelman says.

In August, the Federal Housing Finance Agency asked investors for input on setting up a foreclosure-to-rental program to offload some of the 180,000 repossessed homes held by Fannie Mae and Freddie Mac, the troubled government-sponsored mortgage giants. Barclays Capital, Deutsche Bank AG, Fortress Investment Group LLC (FIG) and Waypoint were among the hundreds of firms that submitted proposals to the FHFA spelling out how investors could participate in such an initiative, according to information obtained by Bloomberg News through a Freedom of Information Act request.

Investors’ Zeal

Wiel says the agency may auction pools of properties to investors, perhaps coupled with federally guaranteed financing that lowers their cost of capital significantly. The Resolution Trust Corp. employed a similar policy in the early 1990s to sell off mortgages held by failed savings and loan banks.

On Feb. 27, the FHFA unveiled a pilot program to sell repossessed houses in Los Angeles, Phoenix, Florida (SPCSMIA) and other hard-hit markets to investors who qualify with the agency. Waypoint submitted an application. “This could be a total game changer for us,” Wiel says.

It’s striking that Washington is looking to Wall Street for answers after investors’ zeal for subprime mortgages helped foment the housing morass, Green says. She says a boom in rentals may encourage mortgage lenders to foreclose on delinquent homeowners instead of reworking their loans to be more affordable.

Shoe-Leather World

Still, she says, leasing defaulted houses does reduce the corrosive impact they have on communities. “Nobody wins when houses are empty,” Green says.

Wiel and Brien, both graduates of the University of California, Berkeley, met at an angel investing conference Wiel was hosting in San Francisco (SPCSSF) in 2008. They talked about the housing crash and agreed that plunging property values in the Bay Area’s bedroom communities presented an irresistible opportunity. So they set up a company with $1 million of their own money and acquired 26 houses during the next six months.

From the outset, the duo applied technology to a business rooted in the shoe-leather world of appraisals, home inspections and foreclosure sales on courthouse steps.

“We asked, ‘How do we systematize and automate everything? How do we scale?’” says Wiel, an upbeat man who’s fond of techie lingo. By 2011, they had hired almost 100 employees and raised more than $90 million from investors in seven funds. It’s midmorning on Dec. 14, and Waypoint’s office in a downtown Oakland high-rise is bustling with activity.

Most Desirable

In a warren of cubicles, leasing reps sporting telephone headsets talk with potential renters. A half dozen members of the home-acquisitions team are crammed into a bullpen outfitted with a brass bell that’s rung with gusto every time a new house is bought. Doug Pankey, a longtime appraiser who helps run the team, reviews the foreclosures slated for auction this afternoon on two computer screens.

Waypoint uses a combination of its own proprietary algorithms and business software from San Francisco-based Salesforce.com Inc. (CRM) to turn potential acquisitions into rentals. Pankey zeroes in on a three-bedroom residence in a middle-class subdivision of Antioch, a San Francisco suburb of 102,000 residents.

The house appears in the center of a red, pulsing orb on a Waypoint heat map that highlights the town’s most-desirable blocks. The house needs $20,000 in renovation, has no liens and earns a 92 out of 100 on Waypoint’s Geographic Scoring System.

Home Rescue

This proprietary program ranks potential acquisitions by factoring in location, proximity to freeways and commuter trains and the home’s historical property-value performance. Pankey watches as the program calculates that with a maximum bid of $150,293, Waypoint can rent the residence for $1,799 a month and earn a 7.7 percent annual return.

Pankey’s buyer, Ken Major, is looking at the same Antioch house profile on his iPad outside the Contra Costa County courthouse, 27 miles (43 kilometers) away, ready to bid. Waypoint maintains all of its property profiles on an online cloud database so agents in the field and supervisors at headquarters can access and update them in real time.

A month after buying the house for about $2,700 less than that maximum bid, Waypoint outfitted it with a new kitchen, carpeting and landscaping.

Family Pictures

Many of the firm’s conversions don’t go as smoothly. On a warm January morning, James Gordon sets out to visit almost a dozen Waypoint houses that may still be occupied. Gordon, a former mortgage broker, is a home rescue specialist who negotiates with occupants to determine whether they can be converted into renters, paid $1,000 to move out or be evicted.

About a third of Waypoint’s homes are occupied by the former homeowners themselves, with one out of four staying on as tenants. Waypoint offers to set aside a percentage of any tenant’s rent so that money can later be used toward a home purchase.

Most of the time, occupants have to leave within 15 days of Waypoint’s purchase because they can’t afford the rent or choose to go. Gordon returns to one residence where a family has refused to move out for six months as they pursue a legal claim that they’re the victims of mortgage fraud. No one’s home, but two brand-new radio-controlled toy cars sit under the Christmas tree and family pictures line the mantle. Gordon sighs. It’s going to take more time before Waypoint earns a return on this property.

Fiscal Stress

At another house, a woman refuses to open the door for Gordon. It turns out she’s one of six different tenants renting rooms there. Speaking through the door, Gordon says she may be able to stay on as a Waypoint renter, but she rebuffs him.

“I’m not trying to be a bad guy, but if you don’t come to an agreement with us, we’ll have to move forward with the eviction process,” Gordon says as he slips his business card under the door.

For all of the cloud computing, the business of converting foreclosures into rentals is often about dealing with households under enormousfiscal stress. Waypoint employs former financial counselors from nonprofit organizations to help tenants repair their credit and even set up household budgets so they don’t fall behind in their rent.

As Waypoint triples the number of houses it buys daily and expands in California and possibly Nevada, Arizona and Illinois, it will have to hire dozens of appraisers, leasing agents and other personnel. The firm will have to shoulder these upfront labor costs before its new funds can earn a profit.

New Industry

“That pulls a lot of inefficiencies into the strategy and that can be expensive,” says Chris Hentemann, managing partner of 400 Capital Management LLC, a New York-based hedge fund with $350 million in assets that invests in mortgage-backed securities.

In April, Waypoint launched a fund to reboot foreclosures in Southern California, and in the third quarter, it recorded $324,327 in operating expenses on $23,920 in rental revenue.

A few days before Christmas, the mood is festive at Waypoint’s holiday party. Wiel and Brien hand out thank-you cards with bonuses as employees pile barbecue, chili verde and brownies onto paper plates. Taking the floor, Wiel says the housing market has only worked through half its backlog of foreclosures and the firm will double its head count as it opens new offices.

“We are at the birth of a new industry,” he says to applause.

Worth the Trouble

Such enthusiasm can be found at many startups anticipating explosive growth. While Waypoint’s strategy is now drawing interest from Wall Street and Washington, Wiel and Brien still must bring order to the inefficient business of turning around foreclosures. And they’ll have to show investors that the endeavor is worth the trouble.

Researchers: Having too many connections weakens networks

By Joab Jackson

When it comes to connecting networks or other systems together, it is best to have many, but not too many, connections, mathematicians have found.

Administrators and network engineers have long assumed that the more connections they insert between multiple networks, the more resilient the communications between these networks will be. The Internet, for example, derives much of its resiliency from multiple, redundant links. But this is true only up to a point. Too many connections can actually be dangerous, because failures in one network can easily cascade to the other, noted Charles Brummitt, a mathematics researcher at the University of California, Davis, who led a team that looked into this issue.

Instead, network owners should fine-tune the number of connections for maximum resiliency, Brummitt said.

Brummitt’s team published its work in this week’s issue of the “Proceedings of The National Academies of Science.”

The work is a mathematical model of how a collection of systems works together. “We’re taking a larger view and studying networks of networks,” he said. Interconnected networks can be vulnerable to cascading failures, in which a failure, or overload, in one network can disrupt another network. In a typical scenario, when one network is overloaded, it will offload its traffic to the second network. But if a failure is enough to overwhelm the first network, it may overwhelm the second network as well.

“There are some benefits to opening connections to another network. When your network is under stress, the neighboring network can help you out. But in some cases, the neighboring network can be volatile and make your problems worse. There is a trade-off,” Brummitt said. “We are trying to measure this trade-off and find what amount of interdependence among different networks would minimize the risk of large, spreading failures.”

The study, also available in draft form (PDF) at ArXiv, primarily studied interlocked power grids but could apply to computer networks and interconnected computer systems as well, the authors note. The work could influence thinking on issues such as how to best deal with DDoS (distributed denial-of-service) attacks, which can take down individual servers and nearby routers, causing traffic to be rerouted to nearby networks. Balancing workloads across multiple cloud computing services could be another area where the work would apply.

“As a first theoretical step, it’s very nice work,” said Cris Moore, a professor in the computer science department at the University of New Mexico. Moore was not involved in the project. “They found a sweet spot in the middle,” between too much connectivity and not enough, he said. “If you have some interconnection between clusters but not too much, then [the clusters] can help each other bear a load, without causing avalanches [of work] sloshing back and forth.”

Of course, one of the largest networks of networks is the Internet. For the Internet, backbone providers peer with one another, or connect their networks together, which allows Internet traffic to move seamlessly from source to destination. Much has been made about the Internet’s natural resiliency in the face of disaster. But is it as resilient as it could be?

“That’s a thorny question,” Brummitt admitted. “I don’t think we are in a position to make any guesses. Even understanding the network structure of the Internet is a problem in itself. But the Internet has proved to be rather resilient. So far, it seems like the Internet is not too interdependent. But this is speculative.”

Apologists for rising inequality often argue that since most Americans’ income has risen despite rising inequality, there’s no reason to complain about inequality other than envy. So it’s worth remembering that we used to expect economic growth to deliver large increases in real income, not just a bit of a rise that’s accomplished in large part through longer working hours; and that a major reason so many have seen such small gains is that a large part of growth has been siphoned off to the very high end.

Lane Kenworthy had a nice chart illustrating both points, comparing median family income with real GDP per family (for those worried about the fine points, it was nominal GDP divided by the CPI, avoiding some technical issues):

You see the contrast: a doubling of family incomes in the post war generation compared with maybe 20 percent since, and family incomes growing in line with GDP before, lagging far behind since, with the difference basically being the rising share of the 1 percent.

This is real stuff, not some trivial envy-driven concern. But we must be very, very quiet about it, right?

[Doing my best imitation of Hugh Grant] Riiiiight…

I am a PHP developer

• I am not a Zend Framework or Symfony or CakePHP developer
• I think PHP is complicated enough

I like building small things

• I like building small things with simple purposes
• I like to make things that solve problems
• I like building small things that work together to solve larger problems

I want less code, not more

• I want to write less code, not more
• I want to manage less code, not more
• I want to support less code, not more
• I need to justify every piece of code I add to a project

I like simple, readable code

• I want to write code that is easily understood
• I want code that is easily verifiable

Nicely put…