tag:blogger.com,1999:blog-18448298038623352522024-02-20T14:07:59.300-05:00MySQL-PythonDeveloper blog for MySQL Python projectAnonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.comBlogger18125tag:blogger.com,1999:blog-1844829803862335252.post-68947286470261086142012-11-16T12:18:00.002-05:002012-11-16T12:18:46.460-05:00Is MySQLdb hard to install?The short answer is: No. The longer answer is: It depends. With the latest version, most of the time, you can run:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">pip install MySQL-python</span><br />
<br />
and you're done... if you have the prerequisites. That's where it gets sticky.<br />
<h2>
The dreaded prerequisites</h2>
Since MySQLdb uses a C module to link against MySQL's client library, you will need:<br />
<ul>
<li>A C compiler and associated toolchain</li>
<li>Python development headers and libraries</li>
<li>MySQL development headers and libraries</li>
</ul>
<h2>
Linux</h2>
MySQLdb was originally developed specifically for use on Linux, as it was the only platform I cared about at the time. Over time, several Linux distributions (Red Hat, Debian, Ubuntu, etc.) picked it up, created their own packages, and distributed them. If you are a Linux user, I recommend you use your distributions packages. But what if you can't, and you really do need to build your own? Install using pip. Each Linux distribution has packages for all the prerequisites, but the names of these packages vary. For Ubuntu (and probably Debian):<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">apt-get install build-essential python-dev libmysqlclient-dev</span><br />
<h2>
Other UNIX-ish</h2>
MySQLdb has been a FreeBSD port for 12 years. Use it. Hey, if you're using FreeBSD or another *BSD, or Solaris, you probably don't need my help to install it...<br />
<h2>
Almost UNIX: Mac OS X</h2>
Mac OS X is very POSIX-like inside, and it so MySQLdb's setup treats it as POSIX, but it can get complicated. The simple part is: You need <a href="https://developer.apple.com/xcode/" target="_blank">Xcode</a>, which is Apple's compiler toolchain.<br />
<h3>
The easy way</h3>
By far the easiest way to get MySQLdb on the Mac is to use one of the two major UNIX-ish package distributions: <a href="http://www.macports.org/" target="_blank">MacPorts</a> or <a href="http://mxcl.github.com/homebrew/" target="_blank">Homebrew</a>. Of the two, I have only used MacPorts. Once upon a time, everything in MacPorts had to be built from sources, like a *BSD or Gentoo-style port system, but there are now binary packages for many things. Some people swear by Homebrew. Either one will make your life easier. MacPorts has a port for MySQLdb (py-mysql and variants). If Homebrew has a "formula", I can't find it, but a pip install should work once you have compatible versions of Python and MySQL.<br />
<h3>
The hard way</h3>
The hard part about the Mac is, there are three architectures you could be building for: PowerPC, Intel 32-bit, and Intel 64-bit. We can safely ignore Motorola since Apple hasn't used them in a very long time, and PowerPC can be mostly ignored, but you still see some support for it. For example, the Python Apple distributes in Snow Leopard is a fat binary that includes code for PPC, i386, and x86_64. You might think this makes things easier, but it does not. If you're on any sort of modern Mac, it's going to use the x86_64 support. That means the version of MySQL you use also has to be built for x86_64. You can't mix and match.<br />
<br />
Most people who are going to use Xcode instead of MacPorts or Homebrew are probably going to download binary packages of MySQL from <a href="http://dev.mysql.com/downloads/mysql/">mysql.com</a>. They have separate 32-bit and 64-bit packaging, i.e. no fat binaries. You almost certainly want 64-bit packages. Another option is to download their <a href="http://dev.mysql.com/downloads/connector/c/">Connector/C</a> package, and here you will find 32-bit and 64-bit Intel <b>and</b> PowerPC as well. You could also download the source and build it yourself.<br />
<br />
Once you have MySQL on your Mac, make sure you can run <span style="font-family: "Courier New",Courier,monospace;">mysql_config </span>from a shell. If it's not on your executable PATH, you'll have to hack on site.cfg to tell setup.py where it is.<br />
<br />
One issue I have seen recently was ultimately an <a href="https://sourceforge.net/p/mysql-python/discussion/70460/thread/43053990/">architecture mismatch problem</a>, and even once that was fixed, the loader path had to be fixed in the environment, which is not ideal.<br />
<h2>
Windows sucks</h2>
Even though Windows sucks, current versions of MySQLdb build on Windows without a lot of drama.<br />
<br />
For the build toolchain, you must have <a href="http://www.microsoft.com/en-us/download/details.aspx?id=14597">Visual Studio 2008</a>, and specifically you need vcsetup.exe for the C compiler. It has to be the 2008 version because that's the version the python.org Python packaging uses. (Of course, if you're building your own Python, you're on your own here.) Like with the Mac folks, you have 32-bit and 64-bit options. Choose the same one for <a href="http://dev.mysql.com/downloads/connector/c/">Connector/C</a>. If you are building a 32-bit package, it should work, period. 64-bit I'm not so sure about. But I've been building 32-bit MSI packages, and pip install should grab one of those if you're using a 32-bit Python, and that is really all you need.<br />
<h2>
Conclusion</h2>
Is it simple? Yes, for most people. Most of the hard work has done for you. So if it's so simple, why don't I have more binary packages? Mainly because, there are several different versions of both Python and MySQL that you might choose to use at any given time, depending on your needs, and you might still choose 32-bit or 64-bit architectures. It's just not practical for me to build all of them. Maybe with some kind of a build-bot. This could change in the future, but for now, there are plenty of other open source package distributors out there, and if at all possible, you should just use a pre-built package.Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com5tag:blogger.com,1999:blog-1844829803862335252.post-90545252259066892242012-11-02T12:06:00.000-04:002012-11-02T12:06:25.360-04:00MySQLdb-1.2.4 release candidate 1I've finally about finished up 1.2.4. Please give 1.2.4c1 a try. I plan to make the final release next week sometime.<br />
<br />
1.2.4 will support Python 2.4, but it's not tested, and won't be supported in 1.3.0. Python 2.5, 2.6, 2.7 and PyPy are all tested and supported.<br />
<br />
I still need to make some documentation updates. It turns out I can't easily make it work with Read The Docs since it has a C module, but I'm planning to have them online at packages.python.org.Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com4tag:blogger.com,1999:blog-1844829803862335252.post-87649688866612774192012-09-24T21:32:00.000-04:002012-09-24T21:32:19.882-04:00A brief history of MySQLdbIt was recently pointed out to me what a confusing mess the source repositories were for MySQLdb, so I spent a good bit of time cleaning them up. To understand how things got into the state they were in, and where we are going, a brief history lesson required.<br />
<br />
I first started working on MySQLdb in 1998. At the time, the best option for source control was CVS, and I had a local CVS repository for the project, and occasionally put up some tarballs. In 2001, I got the OK to make it an open source project, and development moved to SourceForge. Whatever you might think of SourceForge nowadays, it was a safe place to host your code, distribute files, and you got a couple nice extras like a bug tracker and forums. And SourceForge had CVS, of course, but nothing else at the time. Subversion existed in early form but was not yet available at SourceForge.<br />
<br />
SourceForge was a bit slow to pick up on Subversion. Even though Subversion 1.0 was released in 2004, it looks like they didn't make it available until 2006. I do remember that both MySQL and Python both used SourceForge for CVS repositories, and both left SourceForge not too long before it became available, because they just couldn't stand to wait any longer. In any case, once Subversion because available, I switched to using it, because it sucked a lot less than CVS. But one side effect of this was the copying of CVS's branches and tags. I could rename branches and re-tag things, but the old CVS branches and tags were still in there for good.<br />
<br />
In 2008, sometime after 1.2.2, I was ready to make a clean start on MySQLdb, and re-engineer most of it. By this time, Mercurial was available and I decided to port just the SVN trunk over to a new Hg repository. I really had hoped to stop using the SVN repository altogether, but it was still being used for ZMySQLDA, and it also turned out a 1.2.3 bugfix release was needed.<br />
<br />
Work has been sporadic on MySQLdb-2.0 over the last years. Part of it not having a lot of time, and part of it was developer burnout. But there is stuff that needs to be done. In particular, Python 3 compatibility is needed now more than ever. I thought I could do this in 1.2.4, but I would have to sacrifice compatibility for Python < 2.7. So MySQLdb-1.2.4 will be a bugfix release and fully Python 2.7 compatible (and should be Python 2.8 compatible), and very soon thereafter there will be a 1.3.0 which will require Python 2.7 or newer and be compatible with Python 3.<br />
<br />
One parallel development over the life of MySQLdb has been the life of MySQL itself. First they were sold to Sun, which I was not very happy about, but life went on. Then Sun was bought by Oracle, which I'm not happy about at all. Still, since then, you can still get MySQL for free, so it hasn't been huge impact on me. The Oracle acquisition led to two new MySQL forks: MariaDB, led by original MySQL creator Michael "Monty" Widenius; and Drizzle, led by MySQL guru Brian Aker. MariaDB is a straight-up fork of MySQL-5.1 and is supposed to be a binary replacement for MySQL, so the current MySQLdb should work with it just fine. Drizzle, on the other hand, has been heavily refactored and converted from C to C++, and has a different API. There is a Drizzle Python project (which borrows heavily from MySQLdb in places) already.<br />
<br />
Anyway... My plan is to be able to support MySQL, MariaDB, and probably Drizzle, which is going to require having different driver backends. Not so much for MySQL and MariaDB, but definitely for Drizzle. One of the other drawbacks of MySQLdb-1.x is, although there's some support for using the embedded server library, it's hard to have both the embedded server and the regular client library in the same installation. Additionally, building the C module on Windows has historically been painful, and there have been requests for a pure Python connector. I've also considered doing a ctypes-based driver; at the time, ctypes was not ready for prime time, but it is in the Python core for quite a while now.<br />
<br />
With all these various back-end drivers possible, and support no longer being strictly limited to MySQL, I decided it would be best to rename MySQLdb-2.0, and for better or worse, the name I came up with was... Moist. Why, oh why "moist", you ask? Well, MySQL has the dolphin, MariaDB has the seal, and Drizzle has.. .the drizzle... all things that are wet, or perhaps moist. I thought about "Soggy" but there was already a project by that name on GitHub.<br />
<br />
So: The SVN repository on SourceForge lives on, but only for ZMySQLDA: It's been migrated to a git repository (MySQLdb1) that is synced between SourceForge and GitHub, and this is where MySQLdb-1.x will live. The Hg repository on SourceForge that was for MySQLdb-2.0 has now been cloned to become moist on GitHub. The Hg repository will probably be disappearing in the near future, once I'm convinced everything is OK with the moist repository. I've also added some READMEs to all the repositories with a map of where all the code is buried.<br />
<br />
Any questions?Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com13tag:blogger.com,1999:blog-1844829803862335252.post-87944048530209031392012-09-20T09:44:00.000-04:002012-09-20T09:44:48.479-04:00MySQLdb book and other newsDid you know there is a book specifically addressing how to use <a href="http://sf.net/projects/mysql-python" target="_blank">MySQLdb</a>? <a href="http://www.packtpub.com/mysql-python/book" target="_blank">MySQL for Python</a> was published by <a href="http://www.packtpub.com/" target="_blank">Packt Publishing</a> in September 2010. If you're worried about two years being an eternity on the internet, don't: There hasn't been a new release of MySQLdb since the book was published.<br />
<br />
Speaking of new releases, I built a Windows installer of MySQLdb-1.2.3 for Python-2.7 and using the MySQL Connector/C 6.0. This should be able to connect to any modern version of MySQL (4.0 through 5.5 and newer), and as I understand it, it should also work with <a href="http://mariadb.org/" target="_blank">MariaDB</a>, but this is not yet tested.<br />
<br />
Additionally, there is a 1.2.4 release coming out in the near future which will be a bug-fix release only. It will support Python-2.4 through 2.7 (and should be compatible with Python-2.8 when it arrives). There is some work towards Python-3 compatibility (specifically, 3.2.3), but unfortunately, I can't easily support Python < 2.7 and Python >= 3 at the same time. So after 1.2.4 is out, there will be a 1.3.0 release shortly thereafter which will be for Python-2.7 and 3.2 or newer (and probably earlier).<br />
<br />
Longer term: I've also been working on a MySQLdb-2.0 version which is almost completely rewritten from scratch on the Python side, and modularized (and some code removed) on the C side. It's going to have pluggable drivers, with the intention of having the option of using libmysqlclient (the standard C library), libmysqld (the embedded server library), some kind of pure Python driver, and eventually a driver for <a href="http://www.drizzle.org/" target="_blank">Drizzle</a>. MySQLdb-2.0 is actually going to be renamed <a href="https://github.com/farcepest/moist" target="_blank">Moist</a>, and no, I don't yet have the code in <a href="http://github.com/" target="_blank">github</a> yet, but it will be what is the <a href="http://sourceforge.net/p/mysql-python/mysqldb-2/ci/566baac88764ac28a6dbe3ecde56c228a434fb22/tree/" target="_blank">MySQLdb2</a> repository on <a href="http://sourceforge.net/projects/mysql-python/" target="_blank">SourceForge</a>.<br />
<br />
Is the project leaving SourceForge? It remains to be seen. Every time I get fed up and I'm sure I'm about to move, they have some big update which makes things better. The current location is pretty well-known, and the project has been there since 2001. I do think I will be mirroring a git repository between SF and github at the very least; all the cool kids are on github these days.Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com4tag:blogger.com,1999:blog-1844829803862335252.post-62864673562363597992010-02-22T17:11:00.004-05:002010-02-22T18:28:24.883-05:00Asynchronous programming and MySQLdbIt was asked in the previous post whether or not there would be better async support in MySQLdb-2.0. The answer is a qualified yes.<div><br /></div><div><div>libmysqlclient (the MySQL C client library) has blocking calls, and doesn't have a true async interface. In a nutshell, the three blocking calls that are most important are mysql_query(), mysql_store_result() or mysql_use_result(), and mysql_fetch_row() (sometimes).</div><div><br /></div><div>The original design of MySQLdb uses mysql_store_result(), which stores the entire result set in a MYSQL_RESULT structure; in this case, mysql_fetch_row() does not block. To save memory, the result is immediately converted to Python types and then freed. </div><div><br /></div><div>An alternative cursor implementation (SSCursor) uses mysql_use_result(), which does not internally buffer the result set; this does cause mysql_fetch_row() to block, however. A further complication is that no new queries can be issued on the connection until the entire result set has been fetched. This is the primary reason why mysql_store_result() is used by default, because overall it causes less problems.</div></div><div><br /></div><div>The main issue with using mysql_store_result() it can consume a lot of memory the result set is large. There's still the option of using a LIMIT clause, but it's inconvenient. But mysql_use_result() has it's own problems, in that you have to be careful to cycle through the entire result set before issuing another query. Otherwise you will get the "commands out of sequence" error.</div><div><br /></div><div>I am pretty sure that inside the C client library, the wire protocol is exactly the same, and mysql_store_result() is just pre-buffering everything.</div><div><br /></div><div>So in the end, anything using the C API is going to have to deal with some blocking calls. However, there are some design changes that will make some of these limitations a bit easier to deal with.</div><div><br /></div><div>First, all the various Cursor classes are going away, and there will be only one True Cursor.</div><div><br /></div><div>By default, cursors will still use mysql_store_result() on most queries, because there is a lot that it works very well for. This includes INSERT, UPDATE, and DELETE, which do not return any rows, but also some meta-queries such as SHOW WARNINGS, SHOW TABLES, etc. which do return rows but always a relatively small number, and don't take a long time to execute.</div><div><br /></div><div>SELECT statements, on the other hand, will be detected, and those queries will use mysql_use_result() instead so that rows are not buffered in the C client. They will only be fetched upon demand.</div><div><br /></div><div>The irony of using mysql_use_result() on SELECTs is, you can't scroll (mysql_data_seek()) on the result set, and this is the primary use case for scrolling. However, I still expect to make scroll work, possibly in a limited way, because these cursors will still be somewhat buffered. There will be a user-configurable maximum row limit that will be buffered, and once that buffer is filled, the oldest rows will be discarded. This limit will probably be 1 row by default.</div><div><br /></div><div>How will the "commands out of sequence" error be avoided? If another cursor is created, the first cursor will be flushed, i.e. any remaining rows in the result set, and any additional pending result sets, will be read so that the query can be issued. There is also a clear method for the cursor, which also reads all the rows, but instead of buffering them, it discards them. It also avoids doing any Python type conversion.</div><div><br /></div><div>So if you've gotten this far, you're probably wondering: How does any of this affect asynchronous use? Well... it doesn't. The C client library just isn't designed for that sort of thing. But there is still hope.</div><div><br /></div><div><div>I've wanted a ctypes implementation that would still use the libmysqlclient library but wouldn't require compiler tools to build (and hopefully shut up Windows users). At the time, ctypes was still very early, and had not made it's way into the Python standard library yet (included as of 2.5). Fortunately, Jason Coombs has a patch against MySQLdb (<a href="http://pypi.python.org/pypi/jaraco.mysql">jaraco.mysql</a>) which implements this. The reason I haven't integrate this patch it would replace the original driver, instead of being an option. MySQLdb was never designed for multiple drivers. This has made it pretty close to the top of my TODO list for MySQLdb, because there's another case where this would be useful: There currently is no way to have _mysql (the current C driver module) built against libmysqlclient and libmysqld (the embedded server library, which uses the same API) at the same time without using virtualenv or something like that.</div><div><br /></div></div><div>OK, so now the async people are asking: So how does this help async programming again? Well... it still doesn't. To truly have a non-blocking async driver probably means there will need to be a new implementation that is designed with that in mind. A pure Python implementation could do this. I have a start on one which I got from Monty Taylor of MySQL some time ago. It's not asynchronous either, but could be made so. At least it looks like a good starting point.</div><div><br /></div><div>But here's the thing: As far as I know, there is no standard database API for Python which supports asynchronous operation. And it seems like there should be one. Maybe it's time for <a href="http://www.python.org/dev/peps/pep-0249/">PEP-249</a> to be extended for an asynchronous API. Otherwise every database implementor is going to end up doing their own thing, and it sounds like there is a need for this sort of thing.</div>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com6tag:blogger.com,1999:blog-1844829803862335252.post-20866255670518158332010-02-22T12:48:00.002-05:002010-02-22T13:06:11.445-05:00New releases coming soonKyle Vanderbeek is going to take over as release manager for MySQLdb-1.2. We should have one more release candidate of 1.2.3 first, followed quickly by the final release.<div><br /></div><div>Development on MySQLdb-2.0 has been progressing, and has recently moved to a Mercurial repository on SourceForge. This was imported from the SVN trunk. If you pull from the SVN trunk in the future, you may be disappointed. </div><div><br /></div><div>2.0 is turning into a very major rewrite. I should have an alpha release soon. For now, the hg repository builds and passes all tests, but there are probably a few things that aren't thoroughly tested yet, particularly scrolling on cursors. I'll post more detail along with the alpha release.</div><div><br /></div><div>Python3 support is not immediately in the works, and I probably won't work on it until I am close to a beta. At this point, I would target Python-3.1, maybe 3.2. 3.0 would probably work too.</div>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com4tag:blogger.com,1999:blog-1844829803862335252.post-72509851189983278892009-03-19T11:43:00.002-04:002009-03-19T11:57:49.685-04:00MySQL-python-1.2.3 beta 2 releasedI released the <a href="http://sourceforge.net/project/showfiles.php?group_id=22307&package_id=34790&release_id=668047">second beta of MySQLdb-1.2.3</a> over the weekend. So far I've gotten a fair number of downloads but not a lot of feedback. I did find out though what small tweaking is required to <a href="http://sourceforge.net/forum/forum.php?thread_id=3108914&forum_id=70460">build on Windows</a>. It's also in the <a href="http://pypi.python.org/pypi/">Python Package Index</a>, so if you can also install using <a href="http://peak.telecommunity.com/DevCenter/EasyInstall">easy_install</a> <a href="http://pypi.python.org/pypi/MySQL-python/">MySQL-python</a>. Once I make the final release of 1.2.3, I'll put up more eggs for fringe operating systems (Mac OS X, Windows).Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com9tag:blogger.com,1999:blog-1844829803862335252.post-69655867568608265812009-02-21T13:15:00.002-05:002009-02-21T13:21:36.208-05:00Sprinting at PyCon 2009I've got a <a href="http://us.pycon.org/2009/sprints/projects/mysql-python/">sprint</a> scheduled now for <a href="http://us.pycon.org/2009/">PyCon 2009</a>. I can only be there for the first day of sprints. If there's enough interest, we can probably find a way to sprint earlier during some of the open space session; or it can continue after I'm gone.<br /><br />PyCon 2010 will be in Atlanta, GA, which is a lot closer to home, but not close enough that I can avoid lodging expenses.Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com0tag:blogger.com,1999:blog-1844829803862335252.post-48202180644221980672009-02-01T15:49:00.001-05:002009-02-01T15:49:24.369-05:00Project Status: Community Participation Needed<div xmlns='http://www.w3.org/1999/xhtml'>John Eikenberry has taken over work on ZMySQLDA for some time now and has released ZMySQLDA 3.1. I don't have any ongoing Zope deployment except some Zenoss, and use MySQLdb directly, so I'm not a good test candidate. Superficially, there don't seem to be any outstanding issues.<br/><br/>Since Python 2.6 and 3.0 were released, there has been a lot of demand for prepackaged MySQLdb for those versions. MySQLdb-1.2.2 seems to throw some warnings on Python 2.6, due to a new Python set type (and the old Set module being deprecated), and some old-style exception usage. These problems should be fixed in the SVN version (MySQLdb-1.2 branch). MySQLdb was originally developed for Python 1.5 so some old crufty stuff is still hiding out in there. There are also some build fixes for Mac and Windows.<br/><br/>MySQL-1.2.3 is not too far off. There are probably a couple more fixes that need to go into the 1.2 branch. This will almost certainly be the last release in the 1.2 series. One outstanding question is what versions of Python should be supported in 1.2.3? I've settled on no longer supporting Python 2.3, but I won't go out of my way to break it yet (i.e. decorators, generator expressions). Python 2.4 seems reasonable to support. My current development platform (Ubuntu) has Python 2.5.2 so that's what I'm going to test against. I also plan to test against Python 2.6.<br/><br/>Python 3.0 is a different beast. I'm not sure if it will be possible to have a single version of MySQLdb that works on both Python 2.x and 3.x yet. I'm assuming not. I don't think I'll have Python 3.x support until Python 3.1 at least, though that seems to be not too far off. Guido van Rossum has indicated that Python 3.x will have a faster release cycle than Python 2.x; see PEP-3000. We might even see some early 3.1 releases by PyCon 2009.<br/><br/>MySQLdb-1.3, the precursor to 2.0 and the SVN trunk, is on hold for right now. Some of the fixes that are in the 1.2 branch need to be backported. Currently it does not do any type conversion; this has been removed from the C module and will be done in Python. I don't expect this to adversely affect performance. A lot more will be done using generators and more modern Python-isms. The question here is do I support Python 2.x or only Python 3.x. I'm inclined to say I'll develop it now for Python 3.x, and save backporting for later.<br/><br/>MySQL-5.1 was finally released. There are no big C API changes that I've seen (maybe no small ones either), so you should be able to build against it. For that matter, if you have MySQLdb built against the 5.0 client, I expect it to work fine with a 5.1 server, so save yourself some trouble and don't upgrade if you don't have to.<br/><br/>Similarly, if your OS vendor supplies packages for MySQLdb, use them. MySQLdb is known to be in a lot of Linux distributions, including Fedora, RHEL, Debian, Ubuntu, Gentoo, and others. It's also in the various *BSD distributions.<br/><br/>If you are running Windows or Mac OS X, I still don't use those platforms so I won't be building my own packages for them. If you want to contribute your own packages, I'll host them on SourceForge. Make sure you specify what version of Python and what version of MySQL it is built against.<br/><br/>I am planning a short sprint at PyCon 2009; it will not go any longer than Tuesday, unless there is a lot of unanticipated demand. In light of this, this is the approximate release schedule:<br/><br/>Late February 2009: MySQLdb-1.2.3 beta 1<br/>Late March 2009: Sprint<br/>Early April 2009: MySQLdb-1.2.3 release candidate 1<br/>Mid-April 2009: MySQLdb-1.2.3 final<br/><br/>In order for a sprint to be effective, it can't be just me. I need people to help find, report, test, and fix bugs. I don't think we need more than two days to do this. I suspect we can get a lot done before the official sprint starts. We can probably make the trunk usable. If I could get one or two committed people at the sprint (not counting myself), I'd be happy.<br/><br/>Lastly, MySQLdb development is entirely funded by donations. Nobody pays me to work on it. I've been working on it for the past 10 years, 8 years on SourceForge. I have to plan development around my work schedule to avoid any conflict of interest or intellectual property issues. For the same reason, I am attending PyCon out of my own pocket. Donations are appreciated, though I could also use some interested developers. One or two have come forward lately; the important thing is that you submit patches through the bug/patch tracker. If your patches are good quality, I'll probably give you write access to SVN.<br/><br/>Feedback is solicited and welcomed.<br/><br/><br/></div>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com10tag:blogger.com,1999:blog-1844829803862335252.post-60769246853214979702008-05-14T23:27:00.002-04:002008-05-14T23:31:14.504-04:00Zenoss Deathmatch<div xmlns="http://www.w3.org/1999/xhtml">I've been working for the last week on implementing Zenoss to replace <a href="http://www.nagios.org/">Nagios</a> and <a href="http://www.cacti.net/">Cacti</a>. Individually Nagios and Cacti are pretty good at what they do, but they don't integrate well.<br /><br />Nagios is primarily an availability monitor, so it's good for notifying you when something goes down, or a disk is filling up, or the load average is too high. etc., but it's not so great for monitoring performance. Nagios 1.4 uses text configuration files. There is a templating system which can be helpful if you have a lot of identical systems.<br /><br />Cacti, on the other hand, is pretty good at monitoring performance, as in how much bandwidth are you using, resource utilization, and so on with nice long-term graphs using <a href="http://oss.oetiker.ch/rrdtool/">RRDtool</a>, but it's not so great for notifying if something is down. Cacti is almost exclusively SNMP-based, and as a result, you can usually just point it at a device through the web interface and it will auto-discover everything interesting. If you have more than a few hundred items to measure, you need to use <a href="http://www.cacti.net/cactid_info.php">cactid</a>, which is a very fast threaded poller written in C.<br /><br />I've been using both for about 3-4 years separately, but because they don't integrate easily (even though both use MySQL as their backend storage), there's a lot of duplication of effort in getting both of them configured.<br /><br />And then there's Zenoss. Zenoss does both availablity and performance monitoring, with long-term graphing using RRDtool, log analysis, and network-based auto-discovery. Zenoss is written in Python using the Zope-2 framework. Most of the device metadata is stored on ZODB, Zope's native object database. Long-term performance data is stored in RRDtool. Event logs are stored in MySQL.<br /><br />Everything in Zenoss integrates together very well. The data is faceted in the sense that you can browse devices by location, by class, by group, or by system. It has a built-in syslog server, it can use WMI for monitoring Windows systems, it has very flexible event handling.<br /><br />There are still some rough edges in 2.1.92, which is a beta for 2.2. First is, it's a bit of a memory hog and I'm inclined to believe there are some memory leaks. After a day or two the main process will start to use over 200 MB; restarting tends to knock it back down to to around 100 MB or so.<br /><br />Syslog support has some issues. When I first started feeding it some syslog data, all the events were being classified as "/Unknown". This is normal. Once you have some log entires, you can then tell it to map that entry to an event. The problem was, the events had components (the process name when parsing syslog data), but they had no event classes set. Looking at the code, it seemed like it should have been setting the event class ID to whatever the component/process name was. It just wasn't. After some Googling, I found out the code to build the event class key was just plain broken. After making these <a href="http://lists.zenoss.org/pipermail/zenoss-dev/2008/001892.html">suggested changes</a>, I could start mapping events.<br /><br />Another syslog problem was in parsing the hostname. I have a satellite syslog-ng server in a remote location that logs to my central syslog-ng server. Because of this, the hostname has the relay information in it. Zenoss' syslog support has an option to parse this though, so no problem, right? Despite turning this on, I was still getting entires like IP/IP, so back into the syslog code. It turns out, Zenoss expects the separator between the two hostnames to be "@", and syslog-ng uses "/'. Easy fix in the code, but I suspect this may work for the standard syslog, and it needs to be a configuration option.<br /><br />Despite all of this, I like Zenoss a lot. I am running it parallel with Nagios until I get all the event handling nailed down. I might need 2 GB RAM on the monitoring server though, and I have already moved the MySQL database onto a different server.<br /></div>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com9tag:blogger.com,1999:blog-1844829803862335252.post-55524342795625735972008-04-08T17:25:00.004-04:002008-04-08T17:28:44.808-04:00CrunchyFrog: A database navigator and query tool for GNOME<div xmlns="http://www.w3.org/1999/xhtml"><p><a href="http://cf.andialbrecht.de/">CrunchyFrog</a> is a database navigator and query tool for<br /><a href="http://www.gnome.org/">GNOME</a>.</p><p>Currently PostgreSQL, MySQL, Oracle, SQLite3 databases and LDAP servers are supported for browsing and querying.<p>I gave it quick try and it looks really promising. Gotta love a Monty Python reference in any case.<br /></p></div>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com0tag:blogger.com,1999:blog-1844829803862335252.post-7160697962556856272008-03-24T16:05:00.002-04:002008-03-24T16:09:11.723-04:00MySQL Cluster support for Django<div xmlns='http://www.w3.org/1999/xhtml'> Ivan Sagalaev has created a <a href='http://softwaremaniacs.org/soft/mysql_cluster/en/'>mysql_cluster</a> database backend for <a href='http://www.djangoproject.com/'>Django,</a> which allows you to configure master and slave servers, and then specify which should be used on given view with <a href='http://www.python.org/'>Python</a> <a href='http://wiki.python.org/moin/PythonDecorators'>decorators</a>. Found via <a href='http://del.icio.us/tag/mysql+python'>del.icio.us</a> and <a href='http://simonwillison.net/2008/Mar/21/mysqlcluster/'>Simon Willison's blog</a>.<br/></div>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com2tag:blogger.com,1999:blog-1844829803862335252.post-52955912390657495642008-03-14T16:05:00.002-04:002008-03-14T16:09:04.938-04:00I am not dead<div xmlns='http://www.w3.org/1999/xhtml'>It's been nearly a year since the last post, so you might naturally wonder if I am dead or stopping development of MySQLdb. Actually I've been sick for the last year and a half or so, and hadn't really been motivated enough to do anything. Here's what's been going on:<br/><br/><a href='http://sourceforge.net/users/eikenberry'>John Eikenberry</a> has taken over development of ZMySQLDA, since I pretty much don't do anything with Zope these days. He's made a couple of releases on the way to a 3.0 release. As far as I know, ZMySQLDA is still only useful with Zope 2 as Zope 3 has a different architecture and comes with MySQL support directly.<br/><br/>Monty Taylor from MySQL AB has volunteered to help out on MySQLdb. I believe the way this is going to work out is he's going to be doing maintenance on the 1.2 branch, and get some minor bug fixes out there. In addition, he has a good start on a native (i.e. written in Python) MySQL driver.<br/><br/>I have a lot of work done towards the first 1.3 version, which will be the development branch (SVN trunk) for 2.0. To a large extent, this is a refactoring project, which means there are a lot of internal changes that don't affect <i>most</i> users.<br/><br/>When I first started working on MySQLdb (way back in the last millenium), there was only one option for talking to the MySQL server: Use the C API via libmysqlclient. Then the re-entrant/thread-safe libmysqlclient_r was added. Then the embedded server libmysqld. And now there is the prospect of a native Python version. This makes building more complicated because you currently have to build against one library at a time. It's also more complicated for users: Suppose you want to have both a regular client version and an embedded version on the same system?<br/><br/>1.3/2.0 is going to fix this by building all the possible options as separate drivers, and then you can specify which one to use at connect-time, or you can use the default, which will probably go in the order libmysqlclient_r, libmysqlclient, and native. (The embedded version requires special initialization so it should never be a default.)<br/><br/>1.2 and earlier uses a type conversion dictionary to map MySQL column types to functions which convert a string result to a Python type. Additionally, this same dictionary is used to convert Python types into SQL literals. I think in earlier (pre-1.0) versions, these were separate dictionaries, and they were later combined because they have a disjoint set of keys. I'm not sure now if this was a good idea or not.<br/><br/>One of the other complications of this approach is TEXT columns. To the MySQL C API, TEXT columns have the same column type as BLOB columns. The difference is the presence of a flag. This took some kludgy stuff to get to work.<br/><br/>Then unicode came along, not just in Python but in MySQL. (The original target versions where MySQL-3.23 and Python-1.5.) This complicated the type conversion because now it was dependent on the connection's character set, which could be changed, so the converter dictionary had to be tinkered with on the fly. Additionally, there were reference count problems (and maybe still are to an extent) with this approach, due in part to the dictionary being about to be overridden by the user.<br/><br/>I haven't decided entirely how this is going to be fixed, but I will have some method for users to override the type conversion at runtime. I will probably have some hooks that will allow you to use a specific conversion for column based on column type, column name, table name, database name, or any combination thereof. For example, you could have a rule that said that any column name ending with "_ip" with a column type of UNSIGNED INTEGER could be returned as a user-defined IP object, but stored in the database as a four-byte integer.<br/><br/>The type conversion from MySQL to Python currently takes place in the low level driver (_mysql). Since there are going to be multiple drivers, this is going to move up into the Python layer. I don't believe this will adversely affect performance. Looking up the right converter is only a dictionary lookup anyway, and only has to be done once per column per query. Once you have a list/tuple of converters for the result set, these can be applied quickly with a list/generator comprehension.<br/><br/>MySQLdb-1.2 and earlier have several cursor classes which are built with mixin classes. The mixins control things like whether the rows are returned as tuples or dictionaries, or whether a buffered or unbuffered result set is used (i.e. mysql_store_result() vs. mysql_use_result()). This is is pretty messy and is going away.<br/><br/>The format of the row will probably be controlled by a hook of some sort. I'm inclined to using unbuffered cursors, i.e. SSCursor or mysql_use_result(), by default. The tricky part is the entire result set must be fetched before another query can be issued, so if there are multiple cursors, there needs to be a mechanism so that only one can use the connection at a time. Rather than locking the connection, there will need to be a way for one cursor to tell the others that they need to buffer the rest of the result set.<br/><br/>Some of this is already done for the trunk, but needs to be committed. In particular, there is no type conversion at all, and the driver selection is not done yet, but I'll see if I have time to work on it more this week at <a href='http://us.pycon.org/2008/about/'>PyCon 2008</a>.<br/></div>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com3tag:blogger.com,1999:blog-1844829803862335252.post-51265031887478663432007-04-30T14:26:00.001-04:002007-04-30T17:42:49.970-04:00MySQL Conference 2007<div xmlns='http://www.w3.org/1999/xhtml'>The 2007 MySQL Conference is over, and I finally made it back home. I have some notes on some of the sessions, which really aren't that great, so if you want to see what you missed, you should read <a href='http://www.planetmysql.org/'>Planet MySQL</a>. But I will give some of the highlights.<br></br><br></br>There's a lot of new development around storage engines.<br></br><br></br>MySQL-5.1 has a pluggable storage engine architecture which allows you to load and unload storage engines while the server is running. Brian Aker explained that this is for cases where you have a stable server setup and only want to upgrade the storage engine. All the storage engines in 5.1 are pluggable, and there are already some third-party proprietary storage engines available.<br></br><br></br>One of the relatively new third-party storage engines is SolidDB. Solid has been around for quite awhile. In fact, I was using Solid for a project in the late 1990's before I started using MySQL.<br></br><br></br><a href='http://www-03.ibm.com/press/us/en/pressrelease/21430.wss'>IBM announced a partnership with MySQL AB</a> to create a DB2 storage engine, but so far this is only on their i5/OS mainframe platform.<br></br><br></br>InnoDB is still very much alive and well, despite being purchased a year and a half ago by Oracle. In fact, InnoDB OY has already renewed their OEM agreement with MySQL AB until at least mid-2009, so there is no danger of current InnoDB users being cut off.<br></br><br></br>Falcon is a new storage engine which will be part of MySQL-6.0 (2008); <a href='http://dev.mysql.com/downloads/mysql/5.2.html'>alpha versions are available now</a>, and a beta is expected later this year.<br></br><br></br><a href='http://www.nitrosecurity.com/products/nitroedb-data-management.asp'>NitroEDB</a> is a storage engine designed to handle heavy insert loads, and for fast aggregate queries. Aggregate values are stored in the index, so many aggregate functions can be evaluated just by looking at a couple of index values.<br></br><br></br><a href='http://scaledb.com/'>ScaleDB</a> uses a <a href='http://en.wikipedia.org/wiki/Patricia_trie'>Patricia trie index</a> which is highly compressed compared to a B-tree. The implementation has three relatively small in-memory index layers on top of the trie which minimized disk access.<br></br><br></br>MySQL-5.1 adds <a href='http://dev.mysql.com/doc/refman/5.1/en/log-tables.html'>log tables</a>: The general and slow query logs can be set (on the fly) to record into tables instead of flag log files. The only supported engines for log tables are MyISAM and <a href='http://dev.mysql.com/doc/refman/5.1/en/csv-storage-engine.html'>CSV</a>.<br></br><br></br>Coolest presentation: Maybe <a href='http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10931'>The Declarative Power of Views</a>. Basically, this is using SQL like <a href='http://en.wikipedia.org/wiki/Prolog'>Prolog</a>, and creating an expert system with a couple of views.<br></br><br></br>Monty Taylor of MySQL AB has wrapped the NDB cluster API with Swig and come up with some APIs for various languages, including Python, and then created a patch for SQLAlchemy so that it could bypass using any SQL for object storage.<br></br><br></br>Jess Balint of MySQL AB has written a pure Python MySQL driver that looks pretty functional. I'll have more on this in my next post...<br></br><br></br>I held a BoF session for Python users. Considering how late I scheduled it, and that it was up against the MySQL Quiz Show, I had a pretty good turnout. Several MySQL-python users actually had vendor booths: Google uses MySQL (mostly 4.0) and Python for some of their (undisclosed) back-end processes. YouTube (now owned by Google) uses MySQL and Python for some of their user profile stuff. <a href='http://www.snaplogic.com/'>SnapLogic</a> has a new data integration project written in Python which is using MySQLdb, and presumably other database backends. I seem to remember the <a href='http://www.nitrosecurity.com/'>NitroSecurity</a> people were using it as well.<br></br><br></br>All-in-all, it was a great conference, and O'Reilly kept us all well-fed. Next conference starts April 15, 2008. I'll have to try to be a little better organized for that one.<br></br></div>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com3tag:blogger.com,1999:blog-1844829803862335252.post-4992708001372078552007-04-09T17:47:00.000-04:002007-04-09T17:51:21.780-04:00Projects which use MySQLdbI'm putting together a page of <a href="http://mysql-python.sourceforge.net/projects.html">projects which use MySQLdb</a>. If your project is not on this list, leave a comment, with a URL and brief description, and I'll check it out.<br /><div class="section"> <h4><a id="frameworks-libraries" name="frameworks-libraries">Frameworks/Libraries</a></h4> <ul class="simple"><li><a class="reference" href="http://dabodev.com/">Dabo</a></li><li><a class="reference" href="http://www.djangoproject.com/">Django</a></li><li><a class="reference" href="http://www.sqlalchemy.org/">SQLAlchemy</a></li><li><a class="reference" href="http://www.sqlobject.org/">SQLObject</a></li><li><a class="reference" href="http://www.turbogears.org/">Turbogears</a></li><li><a class="reference" href="http://www.webwareforpython.org/">Webware</a></li><li><a class="reference" href="http://www.zope.org/">Zope</a></li></ul> </div> <div class="section"> <h4><a id="applications" name="applications">Applications</a></h4> <ul class="simple"><li><a class="reference" href="http://griffith.berlios.de/pages/download.php">Griffith</a></li><li><a class="reference" href="http://diotavelli.net/rattlesnake.html">RattleSnake</a></li><li><a class="reference" href="http://trac.edgewall.org/">Trac</a></li><li><a class="reference" href="http://roundup.sourceforge.net/">Roundup</a></li><li><a class="reference" href="http://biopython.org/">Biopython</a></li></ul> </div>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com4tag:blogger.com,1999:blog-1844829803862335252.post-48802221274268525262007-03-30T14:15:00.000-04:002007-03-30T15:07:11.418-04:00MySQL Conference AgendaHere's my tentative agenda for sessions I am interested in at the <a href="http://www.mysqlconf.com/mysqluc2007/">MySQL Conference</a>:<br /><h4><a href="http://www.mysqlconf.com/pub/w/54/tutorials.html">Monday</a></h4><ul><li><a href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/12316">MySQL 5.1 In-depth</a></li><li><a href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/12080">Real-World MySQL Performance Tuning</a></li></ul><h4><a href="http://www.mysqlconf.com/pub/w/54/sessions.html#Tuesday">Tuesday</a></h4><ul><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/12203"> Using Stored Routines for MySQL Administration </a></li><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10966"> MySQL 5.1's Log Tables</a></li><li><span class="vevent"><span class="summary"><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/11021"> Implementing a MySQL Client Library</a></span></span></li><li><span class="vevent"><span class="summary"><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10317"> Testing the Security of Your Site</a></span></span></li><li><span class="vevent"><span class="summary"><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10418"> Falcon Concurrency Control </a></span></span></li><li><span class="vevent"><span class="summary"><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10907"> Mission Critical Flight Planning Applications at the U.S. Navy</a></span></span></li><li><span class="vevent"><span class="summary"><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/13986"> How to Build a Highly Scalable News Web Site</a></span></span></li></ul><h4><a href="http://www.mysqlconf.com/pub/w/54/sessions.html#Wednesday">Wednesday</a></h4><ul><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/13950"> Clash of the Database Egos</a></li><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/13869"> Versioning Your MySQL Schemas with Ruby on Rails' Migrations</a> (I don't care about Ruby or Rails but the migrations part interests me)<br /></li><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10931"> The Declarative Power of Views</a> or <a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10992"> Using MySQL as Active RDBMS for Surveillance Applications</a></li><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10976"> Running a Top 500 Web Site with LAMP Stack and Commodity Hardware</a></li><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/12492"> MySQL Server Roadmap</a></li><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10417"> Inside Falcon: MySQL's Newest Storage Engine </a></li><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10913"> MySQL Performance Cookbook</a></li><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10799"> For Ticketmaster, MySQL Replication is the Ticket!</a></li></ul><h4><a href="http://www.mysqlconf.com/pub/w/54/sessions.html#Thursday">Thursday</a></h4><ul><li><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/13873"> Implement High Availability with DRBD</a></li><li><span class="vevent"><span class="summary"></span></span><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10984"> The 7 Stages of Scaling Web Applications: Strategies for Architects </a></li><li><span class="vevent"><span class="summary"></span></span><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/12448"> What Happens After You're Scalable: Capacity Planning for LAMP</a></li><li><span class="vevent"><span class="summary"></span></span><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/12456"> ScaleDB: A Scalable and High Performance Transactional Engine for MySQL</a></li><li><span class="vevent"><span class="summary"></span></span><a class="url" href="http://www.mysqlconf.com/cs/mysqluc2007/view/e_sess/10671"> ORM on Steroids: Using the NDBAPI Directly in Your Mapping Layer</a></li></ul>Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com0tag:blogger.com,1999:blog-1844829803862335252.post-50155980917515176122007-03-28T21:19:00.000-04:002007-03-28T21:31:53.532-04:00Some previous MySQL Confererence postsJust for reference, here are some previous posts I did for the 2005 MySQL User Conference:<br /><ul><li><a href="http://farcepest.blogspot.com/2005/04/mysql-uc-day-0.html">Day 0</a></li><li><a href="http://farcepest.blogspot.com/2005/04/mysql-uc-day-1.html">Day 1</a></li><li><a href="http://farcepest.blogspot.com/2005/04/mysql-uc-day-2.html">Day 2</a></li><li><a href="http://farcepest.blogspot.com/2005/04/mysql-uc-day-3.html">Day 3</a></li><li><a href="http://farcepest.blogspot.com/2005/04/mysql-uc-day-4.html">Day 4</a></li></ul>Not that these are particularly awesome or anything, but there are a few travels notes which may possibly be useful if you decide to go this year, since it's in the same location.<br /><br />If you have the time, and you are from outside The Valley, catch <a href="http://www.caltrain.com/">CalTrain</a> and head out to San Francisco.Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com0tag:blogger.com,1999:blog-1844829803862335252.post-33794638554928076352007-03-27T22:29:00.000-04:002007-03-27T22:40:44.545-04:00MySQL Conference & Expo 2007Thanks to a gift from <a href="http://www.mysql.com/">MySQL AB</a>, I'll be attending the <a href="http://www.mysqlconf.com/">MySQL Conference & Expo 2007</a>. I <a href="http://www.mysqlconf.com/cs/mysqluc2005/view/e_sess/6289">presented</a> at the 2005 conference, and found the conference itself to be pretty educational. Plus the food is generally pretty good at the O'Reilly conferences. Of course the real reason people to conferences is for the swag. Now this year's <a href="http://pycon.blogspot.com/">PyCon</a> had some pretty good swag; I got at least six free T-shirts and two Rubik's cubes. So top that.Anonymoushttp://www.blogger.com/profile/13581204403076051529noreply@blogger.com0