Leaving devkitPro

Written by Ash on Filed in brew Tags: scene personal

I’m going to follow this up with another post for homebrew devs about what this means for me, fling, wut and so on, but I figured this should stand on its own. So yeah, this is a post about me and devkitPro.

At the start of 2019, after WinterMute and fincs engaged with me, exjam and orboditilt, Wii U toolset wut got moved under devkitPro’s umbrella. Prior to this, the project had been doings its own thing - a member of exjam’s decaf-emu group, mostly shipped through my own little pacman repository, wiiu-fling. It was built on top of the compiler and libc from devkitPro’s devkitPPC - wut is similar to libogc or libnx in that it adds console-specific functionality to devkitPro’s generic components.

On its face, this was a pretty good move - tighter collaboration would allow devkitPro’s knowledge about how how their distro works to benefit wut and keep up with any major changes without playing catch-up later. Indeed, the collaboration has led to a few contributions - such as the new syscalls interface and linking tweaks around wut’s C++ support, not to mention the necessity of a networking devoptab - though that was more of a “you’ll need to do this” situation.

This is what sold us on the merger initially, as well as the banner of “official support” and a spot in the devkitPro pacman repositories. However, as time’s gone on, I’ve been left wondering if this was really worth it.

You usually hear a lot of flak given to devkitPro’s leadership (or specifically WinterMute) from folks on Twitter, at varying levels of validity, so I want to be clear on what my perspective is here. I contributed heavily to a devkitPro project - arguably a key contributor to wut and their whole foray into the Wii U. I was added to the GitHub orginisation and private Wii U Discord channels, I’ve swapped extensive DMs with both WinterMute and fincs, I regularly conduct code reviews on wut contributions and merge/reject PRs, and a variety of other things in that vein. I’m not trying to toot my own horn here, more make clear that I really gave this a red-hot go and put my all into working with and under them, for about a year and a half. I did everything I could to be a reasonable, helpful contributor, to adapt to their practices while improving on some of the rough spots (like code review).

Another note to make about my perspective is that I co-admin a homebrew collective called ForTheUsers, which provides things like a homebrew appstore/directory, a DNS for Switch users to access the console’s browser, and (importantly to me) community. The 4TU Discord has the majority of active Wii U homebrew devs and other generally nice folks, which I very much appreciate. We’re planning on doing more things with ForTheUsers in future, but for now, those are our relevant projects.

So, I’m going to document the environment that devkitPro created and fostered for its contributors, and what kind of behaviours I had to deal with. I’m deliberately keeping these somewhat vague - I’m not here to screenshot dunk or dig up old receipts, I just want to enunciate the environment I found myself working in, and my personal reasons for making this decision. Without further ado, let’s get into it.

Mistreatment of active contributors

Members of devkitPro’s leadership have regularly bullied wut contributors over the perceived quality or justification of their other projects. This includes the author’s own pacman repository and their contributions to wut before the devkitPro merger, and another contributor’s suite of helper libraries, extending their umbridge to all the third-party projects using them. Usually entirely unprompted, and often with reference to specific portions of code, these projects are often declared “bad” or “hacky” before more detail is given about how the dev is personally hurting some aspect of the scene, and devkitPro leaderships’ disgust at the project’s continued existence. This has happened in targeted DMs, in channels visible to wut contributors (usually accompanied with a ping), and references to the aformentioned disgust have even appeared in their semi-public Discord channels visible to a wide variety of prominent scenefolk.

Mistreatment of new contributors

devkiPro’s leadership occasionally handle issues and pull requests in wut, and each time they’ve done this they’ve treated contributors with irritation and aggression. For example, one developer - new to the Wii U - went to great lengths to resolve a licensing issue in wut, actually getting individual permission from every contributor. However, when they finally submitted a PR with the license change, a member of devkitPro leadership responded passive-aggressively, stonewalling on minor issues, stating the contributor should have known about them, etc. Framing otherwise constructive feedback like this makes the contributor feel subpar and ostracised, which led to that particular developer ceasing contributions to wut. That same contributor, trying to debug a linking issue, had their good-faith comments deleted, without explanation, by the devkitPro GitHub orginisation, all while being chastised for wasting leadership’s time and providing a test case that wasn’t seen as good enough. This all happened in a thread with another new issue reporter. The author has the original emails from GitHub, and devkitPro’s “build logs clogging up the issue” explanation does not account for all the missing comments.

Mistreatment of extended community members

devkitPro leadership has an extremely negative view of ForTheUsers and its projects, particularly the app store/directory, and particularly against a specific member of the admin team, for reasons that aren’t entirely clear - devkitPro leadership cites “incompatible worldviews”. This led to the author pouring immense emotional labour into reconciling the two groups, proposing collaboration strategies, highlighting misconceptions on both sides and organising meetings and opportunities to talk things out. This was repeatedly met with stonewallinging and anger from devkitPro leadership, who appears to feel that the orginisation is a lost cause. This left the author in a highly awkward position, where they were actively contributing to devkitPro, whose leaderhip regularly targeted and attacked the very orginisation the author co-admins, with particular personal attacks on a specific admin.

Concerning attitudes about non-devkitPro projects

When a member of devkitPro leadership perceives an external project as non-constructive, their response is often to want it “gone” in one way or another - for a library to stop being used, for a homebrew application to be removed from circulation. This often gets expressed using violent metaphors - “strangling” an ecosystem or “killing” a project. This language frequently gets used to refer to contributors’ non-devkitPro projects, such as the aformentioned pacman repository and library suite.

Scathing code review

The pattern of disgusted and passive-agressive responses to contributors continues in private channels, both in terms of code review and administrative decisions. The author ended up feeling like anything significant they did had a decent chance of a Discord ping and a midly disgusted rant, which is not a great environment to work under. In another case, a leading contributor had a commit from 3 years ago dug up, with a long and aggressive argument about its validity ensuing afterwards. This, as usual, happened in the wut contributors’ Discord channel.

Lack of delegation

devkitPro uses a manual system for releasing and uploading packages to their pacman repository. To the best of the author’s knowledge, the only people with access to the package repository is devkitPro leadership. This leads to every new or updated package requiring a significant amount of effort from specific people who may be uninvolved in the inception of that package. Often, these updates don’t happen at all, or are delayed after the buildscripts are published. Members of devkitPro leadership frequently cite their workload and valuable time to justify curtness towards contributors, but they also don’t appear to have made any effort to automate, delegate or streamline their workflow - at least nothing visibly on the Wii U side.

Lack of contributor control

devkitPro leadership has established themselves as the primary decision-makers for wut. As they control packaging and version releases, they have veto-power over any changes - they can simply not ship a change to users. This leaves contributors who previously had a strong say in the project’s direction and administrative side as “mere contributors” rather than project leaders. Combined wtih devkitPro’s leadership unwillingness to delegate, their heavy workloads across several consoles and projects, and their general disconnect from wut and the Wii U scene; this leaves the project in a spot where decisions and actions only devkitPro leaders can do are not done. For example, releases - despite wut receiving frequent and significant code changes and improvments, in the full year and a half since the devkitPro merger the only releases created by leadership were beta8 (Mar 20, 2019) and beta10 (Jun 7, 2020). beta9 (May 22, 2019) was pushed through by the author, believing they had permission, only to find that a green light from half of devkitPro leadership wasn’t enough. The author was pretty heavily chastised for this and has not attempted to create a release since. Before the merger, wut releases were being made every few months.

Lack of quality control

Despite devkitPro’s strict approval requirements for contributions to their projects, the parts handled by leadership alone - specifically, the pacman repository - do not appear to have quality controls in place to prevent common mistakes, things like packages marked as PowerPC containing x86_64 binaries. More recently, the new wiiu-sdl2 package has a blatant linking error. This package, created and pushed entirely by a member of devkitPro leadership shipped out broken, automatically overwriting working binaries from fling. When this was pointed out in devkitPro’s semi-public Discord server, the leadership member recognised the error and built a working version, providing a link to it, but the main package repostiory was never updated. The wiiu-sdl2 package is still broken at the time of writing. While mistakes happen, the lack of a staging area or any apparent attempt to address the frequency of broken packages pushed to users is concerning.

These all come together to create an environment that was highly uncomfortable to work in, repulsive to new contributors, obtuse about distribution processes and administrative decisions, and ultimately one where the core contributors were expected to shut up, write code, and let devkitPro leadership handle the rest - which they rarely did a good job of.

I don’t want to work under them anymore. To borrow a metaphor from devkitPro’s leaders, this decision may be making things difficult for myself, like cleaning windows with cats, but devkitPro’s alternative solution involved such caustic and toxic chemicals that I just can’t keep working with them. I believe we can do better without them, and I’m taking action to accomplish that. I still hope to be able to contribute to devkitPro’s other projects - the PowerPC pacman portlibs, for example - and potentially some of that information-sharing could still take place. I’m going to wait and see, as the ball is now in their court.

I’ll be following up with a post about the specific changes coming for wut, fling and homebrew development on my end. Suffice to say, we’re-a-forking. [paw hand v sign (blue) emoji]

Update: Fork isn’t gonna happen. Something else happened in the scene, and I had a odd moment of just… like this isn’t gonna work. I’m investigating what I’m gonna end up doing with my time - I suspect it’ll be another console, strictly app development, something like that. I’ll try to blog more. Infra projects like wiiu-fling are unmaintained and free to a good home, contact me for deets. Hope to see y’all around.

I don't like Disqus, so I've disabled comments for now.