Paul Boddie's Free Software-related blog

Paul's activities and perspectives around Free Software

Python 3: The Feast after the Enforced Python 2 Feature Diet?

Victor Stinner makes the case for OpenStack moving to Python 3 “right now”. As a member of the Python core development community, it should not be a surprise that he might have such a view. However, any large Python project would arguably be better off just waiting for the core developers to acknowledge the obvious: that deliberately breaking compatibility and making needless work for people was a bad idea, and that bringing people over to the supposed future of the Python language will involve building a bridge for them and not just telling them to jump some difficult-to-measure divide while teasing them with goodies, many of which being things that have been deliberately withheld from Python 2.x for the sake of making Python 3.x look better than it inherently is.

I notice that the referenced article claims that “Python 3 is fast”. First of all, PyPy offers much more convincing performance benefits for many programs, and PyPy did not support Python 3 last time I checked. Secondly, CPython 3 had a lot of scope for being faster than it was initially, and I seem to recall that there were many Python 3 migration measures in Python 2.6 that led to performance regressions there as well, so everyone had to suffer from the break in the roadmap while an illusion was preserved to suggest that CPython 3′s relative performance was not that bad in comparison to the 2.x series. Thirdly, stuff like the “computed goto” work could have been delivered for CPython 2.x, but there has been a deliberate policy of denying such new features to CPython 2 over the past few years. (For example, repeated attempts to diminish CPython’s perverse resistance to being cross-built, with patches against CPython 2 supplied and largely ignored or shuffled about over many years, presumably languish now as tickets targeted against CPython 3.)

I can totally understand that the core developers don’t really want to look at a code base that is effectively an archaic version of the one they now work on, but as a consequence of deciding that things must be broken, people need to understand that many improvements which could have reached them in CPython 2 never made it that far, partly as a policy decision and partly because the deliberate break probably makes backporting much harder than it might otherwise have been.

OpenStack may well end up being better on Python 3 because of all those new things mentioned in the article, but getting there may well be harder than it needed to be. If the project does end up benefiting, the exercise can hardly be considered an endorsement of the Python 3 strategy. And putting Python 2 users on a feature diet in order to make Python 3 look like a feature feast is no way to justify the strategy, either.

4 Responses to “Python 3: The Feast after the Enforced Python 2 Feature Diet?”

  1. toto Says:

    Please.

    You know why python2 is on a feature diet? Because it is dead; that’s it.

    There is no point in adding features to python 2, because if people wanted the new shiny things, they would use python 3. That is the whole point of the extended life support of python2: to keep things working for a long time (and a long time it has been) after the introduction of python 3 until the people who wanted to port have done their work.

    If you want to add features to 2, patch it yourself, be my guest, but there is no point.

    Things must be broken, this is the only way of keeping a project sane, otherwise you just keep piling up legacy shit that will come back to bite you later. Yes, companies that have py2.5 code are not happy about this, too bad, paying a software engineer must be a huge burden.

    At this point, most of your py3k articles are FUD (and you are part of the problem by telling people that it’s fine to stagnate because the python devs are the evil ones), I’m sorry. There is no evidence that “because the deliberate break probably makes backporting much harder than it might otherwise have been”; yes, this is a new version, improved internals and stuff, even if some kind python 2 compat had been kept, there is no telling that it would have been any easier to add things to CPython.

    Also,
    >PyPy did not support Python 3 last time I checked.
    Then check again, it’s on its way https://bitbucket.org/pypy/pypy/branch/py3k

    And because I don’t want to make a comment on your previous article, the sentence “if they have not already been alienated by seemingly petty acts of the Python core developers who have sought to punish them for not enthusiastically embracing the project’s latest vision by some arbitrary deadline” is negated by the existence of 2.7 which, imo, shouldn’t have happened at all.

    I thought we killed those absurd discussions about a 2.8 several years ago, but it’s like a zombie, I guess, you can’t really kill it unless you shot it several times, then pour gasoline and set it one fire.

  2. Paul Boddie Says:

    You use the “FUD” term quite freely, but I pointed out that people wouldn’t take so easily to Python 3 five years ago, and the only surprising thing is that it has taken this long for people to realise what the software engineering world already knew: people hate to have to change things for change’s sake.

    A specific point: the deliberate break makes backporting difficult by definition. I don’t know what “evidence” you want to prove this point beyond the fact that Python 2 and Python 3 are different languages and thus taking Python 3 code and running it in Python 2 is not particularly likely to work. The fact that things have been done to Python 3 to alleviate backporting difficulties (and thus add “legacy shit” back) merely emphasises my point.

    Another point: as I wrote, but you will presumably gladly dispute for the sake of argument, there are Python 3 enhancements that could readily be backported to Python 2, and there are also enhancements that probably depend heavily on the reorganisation involved in making Python 3. (And there are also things which probably don’t make any sense to take from Python 3 and apply to Python 2.)

    Since the “new shiny things” have been tied to Python 3 regardless of which category they are in, claiming that Python 2 can’t have them – thus “rewarding” established users with a lack of security fix coverage in certain cases – the division between the old and the new is largely arbitrary: it exists only because a policy decision was taken to make it so. Python 2 is thus only “dead” because someone has decided it must be so, not because it actually is.

    There are areas that can easily be updated for Python 2 without anyone getting hurt. For a long time, I actually advocated work on the standard library rather than the language, mostly because the standard library has been rotten for many years. Although some work has been done to fix stuff up (for Python 3, naturally), the library has always played second fiddle to the interpreter, and thus we now have the delightful situation of a library whose APIs can’t be easily fixed up even in Python 3 because nobody paid it enough attention and because it was largely carried forward just to have something to use with the interpreter. So much for cleaning up the “legacy shit”, but I guess that’s what Python 4 is for, eh? (Or maybe there’ll be some kind of functioning package manager and everyone will ignore the legacy blob installed by default. We can but hope.)

    As for PyPy, I am perfectly aware of the project to make it support Python 3. As I noted five years ago, implementations other than CPython may well end up saving the day, perhaps by offering interoperability that CPython will not. From what I’ve seen, just levelling the playing field for other implementations has taken a lot of unnecessary effort, and I don’t envy those implementers one bit: having to match every last peculiarity of CPython because the core developers don’t believe in having a neutral language standard (something I heard seriously proposed back in 2006, in fact) must really drain their motivation; no wonder people go off and develop separate languages instead.

    And no wonder people go off and use other languages in frustration at the unaddressed inadequacies of Python implementations: I hear that Go has benefited a bit from Python’s core development priorities. You may not like what I have to say, and strangely enough a bunch of established and notable users of Python seem to agree with me either explicitly – they’ve said the same things – or implicitly – by not “rushing” in the last five years to port to Python 3 – but I’m only telling it like it is.

    Maybe those who thought it would be fun to have every newcomer, user and potential user confronted with books, documentation, tutorials on two different dialects of the language might like to reconsider the wisdom of their decision, for it is they who created the fear, uncertainty and doubt all by themselves.

  3. toto Says:

    Yes, I wholeheartedly agree that people hate change (even if it is not necessarily for change’s sake). For established codebases (and I agree that python has tons of those, of very varying quality) it is always a bother to port anything; but for new projects, python 3 is really great and a step up from python2. And I do not see how the str → unicode transition could have been made without breaking any compatibility.

    One of the things that you are not taking into account is that porting and making something backwards-compatible is always a chore, with subtle, minor API changes that break some part of your application. Python 3 breaks things on a larger scale, but at least you know what to look for. I am not aware of much being added to python 3 to simplify porting, apart from u” which, frankly, should either been kept here from the start, or not added at all (as it will break compatibility with earlier py3k versions).

    I am aware that there are some (if not many) enhancements (mainly stdlib though, CPython changes are trickier) that could be ported to 2.7, but as both languages are different, it would have required twice the amount of regression testing, which is painful. But I agree that it is also mostly a political decision, in the hope that people will someday need that feature and migrate.

    I have nothing to say for the standard library, because I have gone into some of its dark corners and agree to you to a point, but the situation is (very) slowly improving, but as API changes are again not backwards compatible, this adds yet another problem (it’s fun upgrading from 3.2 to 3.3 and discovering some functions have been removed).

    A neutral specification of the language would indeed be a great improvement and I would love to see this happen. Nothing to add except that it implies people diverting resources from actual development to specification and formalization, which experiences tells me is very hard to do, especially in a free software project.

    About CCP’s developers blog: “I also understand that upgrading applications like EVE Online was not part of the Python 3 adoption strategy.” and really, from what he tells us, there are little to no tests, the codebase is monolithic, and part of that also depends on a proprietary middleware still using 2.X. One of the big nos for adoption is twisted, which is being worked on too, https://twistedmatrix.com/trac/milestone/Python-3.x.

    I am really impatient about the end of python 2 life support. Maybe some developers will stop using python, maybe many, maybe there will be enough bummed developers to successfully fork it. But at least we will only have one language called python (with all the quirks of different versions across distros still, but well).

    Anyway, thanks for your response, sorry if I am a bit aggressive, it’s because I have a contact that essentially sends me any anti-py3k link he sees (like CCP’s earlier today, or Alex Gaynor’s which I believe is really not relevant), and this tires me a bit, especially 5 years after the fact.

  4. Paul Boddie Says:

    Well, you really want to use Python 3 and embrace a cleaner language, and I don’t blame you at all for that, but I can tell you that even the opportunity to clean it up hasn’t necessarily restored the elegance the language once had.

    Once we got new-style classes (inheriting from a class called object, indeed) to work around supposed deficiencies in the old-style classes and to provide an alternative to __getattr__, along with a super built-in that didn’t quite offer the magic solution it was supposed to, there was plenty of stuff to clean up, but some pertinent questions include whether this has really occurred and whether it was worth doing at the expense of tackling other issues instead.

    I personally feel that Python 3 is a lost opportunity. Some people have gone off and tried to make their own Python-like languages (that certain parts of the community don’t want you to talk about), and it is definitely worth considering what they have tried to offer: Cobra introduces contracts, Boo attempts some kind of limited type inference (both of these unfortunately requiring the CLR/Mono/.NET), MyPy and Wirbel also offer annotations and limited type inference.

    I am very interested in static analysis of Python. Indeed, I even reserve the right to make my own dialect of Python and to use things like Shedskin that are also effectively dialects of the language, but by doing so I don’t think I’d ever claim that I was offering a way forward for everyone expecting an unobstructed path involving Python as people know and understand it. Adding tons of run-time baggage to Python obviously doesn’t help people to inspect and understand their code before running it, nor does it help developers of various compatible implementations, either in delivering something that people can use or in offering improved performance. Both of these things have always been valid concerns about Python regardless of what the core developers claim.

    Instead of just adding more stuff, there could have been a more sophisticated effort at understanding what Python really is: a bizarre suggestion, but categorizing it successfully has probably never been done. To me, Python 1.3 is still essentially Python, albeit a probably rather annoying version that lacks some nice things I use today, and the Shedskin Python-to-C++ compiler also provides a form of Python. Here, the core developers could have refined what Python is and provided something that could have been the foundation for so much more than what Python 3 actually is.

    Thanks for responding to my post and challenging my opinion, though. You should perhaps get your “anti-py3k” contact to justify his stance, too. Maybe you’ll end up understanding each other’s position a bit better… or maybe he’ll just stop sending you those links. :-)