DownThemAll: Latest News

I’ve just made available the source code to the WebExtension at https://github.com/downthemall/downthemall for those curious and maybe even tech savvy enough to run it straight from the code repository. The code repository also has a Readme and a TODO file detailing what to expect.
It is generally working and functional, but I still need to do some additional testing, especially on Windows and Linux and finish up some rough edges and squash some minor bugs (like the “Invalid Date” in the tooltip in the screenshot below).
So it should be a day or two until I will publish a beta version…
I know the wait feels a bit like Duke Nukem Forever by now…. Sorry for that.

Sorry again, but I got a bit overwhelmed with/distracted by other stuff so could work less on DownThemAll! Lite than I anticipated, and I also keep running into limitations/quirks/bugs/idiocrasies of the WebExtension APIs (and sometimes HTML5 stuff as implemented by Firefox).

So, even though I wanted to have something presentable by now, I failed my self-set deadline once more.

While my development version can make *some* simple downloads already, there is still problems with other more complicated types of downloads that need e.g. special handling of headers. And downloading large files is still far from what I expect/want performance-wise.

I’ll stop promising certain dates at this point. I will only promise that I’ll work as quick as possible.

For the curious: I started writing the first bits of code back in late August and worked on it since then (of course not full-time, as it doesn’t pay my bills). Now the “big” to-do items are mostly done, and what’s left is to get all the nitty-gritty details and edge cases right, or at least right enough that I won’t be ashamed about the product.

Initially I planned to have a release version of DownThemAll! Lite (aka the WebExtension) ready for launch of Firefox Quantum. Well, Quantum will be released in a few days from now, and DTA Lite is not nearly ready.

There were some bits that turned out to be more fiddly than I anticipated, then some freelancing work (what actually pays my rent) took longer than anticipated, and then some medical stuff (nothing to worry, but still preventing me from spending the time I originally planned for DTA) and other personal stuff… This all completely made me miss my timeline I planned.
Please bear in mind that essentially DTA Lite, even tho it’s only “lite”, will be more or a less a complete rewrite, even though it will reuse a lot of the concepts.

Therefore, looking at my calendar, DTA Lite will take another month at least. I hope I’ll get to a point where I can release a beta version for early adopters/testers in 2-3 weeks and then 1-2 weeks after that a stable full release.
In the mean time, there is still the full DownThemAll! which can be used with at least the pre-Quantium Firefox (such as the ESR)*.

I’d also really like to rant how stupid and limited most of the WebExtensions APIs are, but I feel I ranted too much already.

Bonus Screenshot
The manager tab, recently, with some demo downloads I used to test how stuff will look like. The design is still a work in progress.

* (technically, it still even works in Quantum Nightly, with some minor changes and a lot of “convincing”, because, again, mozilla’s reason to disable legacy extensions is not a technical need due to Quantum but entirely a political decision at this point)

So, mozilla now decided that tracking all user activity and sending that data to a third party called “Cliqz”, based in Munich, Germany, majority owned by Burda media, a “news” and print (read: mostly tabloid) media group (Bunte, Focus, Chip, Freundin) is a good idea…

I can only shake my head. Clearly Firefox is the “privacy browser” as they claim!!1!

The dumb stuff they pulled in the past just pales in light of this.

Initially they will “test” this “feature” in Germany with 1% of the user base there, they say. It doesn’t sound like it’s exactly opt-in, but opt-out, so make sure your data isn’t siphoned off next time you install/update Firefox.

I made some more progress today, getting some fiddly bits to (mostly) work.

Also, people suggested I join up on Patreon so people can support me there, so here is my page:

I decided to make a “lite” DTA web extension after all.
Of course, it will have serious limitations and far fewer features, but that’s a limitation of WebExtensions and what can you do? Those limitations so far created a bunch of road blocks that will prevent DTA from working correctly in a lot of cases (like the inability to specify HTTP Referers).

On a maybe positive, it should be easy enough to support Chrome too (and Opera, etc?)

This is roughly the progress so far. I also implemented the Add Download (single URL) user interface for the most part, as well as internal queue tracking.

Regarding the actual downloading, I am still undecided how to best proceed there. There is things we could do with blob: “files” to support segmented downloads and proper resuming and a ton of other things, but browser support right now looks fragile at best…

I’ll update y’all once I got a version ready for public testing.

Alright, so today I wrote a response to an email I got from mig, and while writing it, I figured it might be a interesting read for other people too, although it’s basically a huge rant.
If you cannot stomach the use of the f-word, do NOT read on.

So here is what I wrote and then sent to mig and a few other people with similar questions and also a bunch of mozilla mailing lists.
Please note that it was not about changing minds (I given up on that quite some time ago), but more of a cathartic exercise collecting and writing “away” my frustrations.

The response

Hi mig[-1], and everybody who also asked and I BCCed, and whomever it
may or should concern too,

First: the "fucks" are directed exclusively at mozilla - the
organization, not you.

If I CC'ed you and you're now thinking: Who are you even? Valid
question: I develop one of the most popular Firefox add-ons (open source
without profit motivation), and am a decade long mozilla enthusiast,
advocate and volunteer contributor.


> I'm just back from a Mozilla event where i was sorry to hear you were
> giving up on DownThemAll. I don't know the whole story, but in my
> opinion, this would be a shame to leave a 1.25M users audience.

The whole story is basically that mozilla folks are fucking up the
add-on space.

The whole story is that DownThemAll! would need a ton of niche APIs that
mozilla has neither the resources nor the will to spec, implement and
maintain[0].

The whole story is that WebExtensions APIs explicitly are supposed to be
high level APIs, while tons of add-ons actually want, nay need low level
APIs to implement their functionality.
The rational here seems to be "Fuck yall, we consider you too stupid
and/or evil to give you low level access, also we're lazy and not good
with money so we couldn't possibly support low level anyway"
The high level API shit is what's killing the platform, not XUL or
(partial) XPCOM deprecation.

The whole story is that I just finally grew tired of the steaming pile
of utter rotten horse manure that is the mozilla decision making process.

I'll evaluate the list of forks that do exist or will exist once mozilla
pulls the WebExtension switch for real, and see if any of them will be
an alternative to the then deliberately-made-retarded mozilla browser.

I gave mozilla a list of what interfaces DTA source code contains
currently (mozI*, nsI*) either way and other feedback, since they asked.

It is my opinion that it's not me who's leaving a 1.25M Active Daily
Users DownThemAll! audience, but mozilla is abandoning them (and me) and
not just them but also the developers and users of tons of other add-ons
with small and large audiences[1].

I'll keep maintaining (most of) my add-ons for the time being, albeit
with far less enthusiasm, in case mozilla wakes up or some viable fork
comes along, tho.


> As far as i can tell, DownThemAll will be able to run on WebExtensions
> once the missing APIs (mainly file writing) will be integrated, and i
> got the confirmation this will happen in due time.

I have no hopes that they will implement proper APIs, not even for file
writing[0 again]. Other than file writing, there are no proper APIs to
do requests, there are no proper APIs for other stuff such as executing
files, other kinds of OS integration, UI integration and so on and etc
and pp.

And that's just DownThemAll!, looking at my other add-ons (public or for
personal use) and also those I use of other devs, most of them will be
dead in the water, or could only be ported with serious, serious
limitations. Some add-ons I use already were abandoned, rightfully so
because WebExtensions offer no way forward for those addons, and for now
I fix them locally for me if something breaks (I cannot take over
maintainership and publish them as I lack the time and motivation to do so)
I have no use for crappy webrequest/toolbar button APIs alone. At least
the Adblockers will survive I guess... hurray!


Dismantling the add-on system just because mozilla doesn't like the
maintenance burden all of a sudden?

"B-but we want away from XUL and a lot of XPCOM".
So what? Neither is this going to happen anytime soon realistically, nor
is that any reason not to give add-on developers access to whatever
replaces it.

"B-but add-ons will break less if ever if they are WebExtensions".
Sure, and tons of add-ons should and will go the WebExtensions route.
Doesn't mean you have to fuck over the add-ons not fitting in the
WebExtensions space. There are tons of dedicated add-on developers who
have been dealing with breaking changes in Firefox since it first got
add-ons, for better or for worse. Most of the time, we managed in a
timely fashion.

Even those add-ons which can be reasonably ported need to be ported in
the first place. Somebody will have to do the actual work, which is on
entirely different scale than a "few" "let's move this shit into a
framescript so it works with e10s" fixes.

Frankly, it's add-ons which contributed a lot to Firefox' success, and
it's add-on which eased Firefox bleeding users to Chrome, and once the
add-ons that go beyond WebExtensions stuff are gone, the bleeding will
only increase again.


> To tell the truth, i have been myself very frustrated just a few weeks
> ago, and considered giving up VDH on Firefox. Now i can see a clear
> future (even if there is a lot of development work to be done).

Quite honestly, I'm over the frustrated stage and arrived at the furious
anger stage. And I grow only more hopeless about mozilla as time progresses.

WebExtensions are far off from feature parity, let alone bug parity for
even the Chrome extension APIs, yet announce EOL for new add-ons in 53
and EOL for all add-ons in 57 [0 again]?
What the fuck are they thinking?
Whoever was involved in that decision with actual say: Please do us all
a favor and just step down from any leadership position you might have.
Or better yet, apply for a leadership position in the Google Chrome
team; Firefox can use some help from you eventually ending up
inadvertently sabotaging Chrome sooner than later.

What's even more discouraging is that mozilla will be using their
"signing required" Walled Garden they installed because "reasons, not of
them actually sane or good" that they swore they will not use to fuck
with add-ons[2] - just to do exactly that, and fuck with add-ons,
stopping to sign new non-WE add-ons with the Firefox 53 release.
THANK YOU VERY MUCH! I TOTALLY TRUST EVERYTHING YOU SAY NOW MORE THAN EVER!

Does the Walled Garden help make Firefox more secure? Nope.
Is it abused to force unrelated policy changes instead? Yep!



mozilla has been a huge clusterfuck for years now, not just in the
add-on space; lacking proper (tech) leadership, lacking vision, focusing
on the wrong things at large more often than not, fucking with their
core users for no apparent reason other than "but we have to do
*something* to stay relevant". And even stupid stunts like force
bundling crapware (pocket) isn't too goddamn stupid to do these days.
"1 million mozillians!", yeah, you will certainly achieve this by
alienating everybody on many fronts at once.

I've been part of the mozilla universe for almost one and a half decades
(or almost 15 years in "metric") now. I'm doing DownThemAll! and other
extensions since about a decade now. I've seen tons of fuckups in that
time, and produced a few myself; but that was OK because none of those
were deliberate and we always worked together on fixing things.
Not ever before did I think mozilla is hopelessly fucked at a
fundamental level. But the last one or maybe two years changed that.

I have to admit that I failed to see this for a quite some time,
deluding myself into thinking "it's not that bad", "they'll will do it",
"temporary setback", "they will recover", "I can learn to live with
that"... Tried to rationalize all this away...
But that's ended.

I'm fed up as an add-on developer, I'm fed up as a mozilla advocate, I'm
fed up as somebody who used to help the other add-on devs, I'm fed up as
somebody who contributed an enormous amount of volunteer time directly
in many different ways, I'm fed up as a Firefox user.

In conclusion, let me end with two quotes from[3] (second one quoting
myself)

"It’s fascinating how Mozilla manages to always find the exactly right
words — to make their most avid browser enthusiasts feel absolutely
miserable!"

"I honestly hate you [mozilla] right now."
"Bye"

Nils

PS: If anybody feels the inexplicable urge to reply and wants me to know
about it or even respond, make sure to CC me.


[-1] who is on BCC because publicly posting his email address might be rude.
[0] I'm explicitly not dumping on the team that actually implements the
WebExtensions support and APIs, they seem to be doing a fine job with
the resources they got from mozilla. And I am not opposed to
WebExtensions, quite the opposite. But I am opposed to WebExtensions-only!
[1] Well, unless you're NoScript and get special treatment. Well again,
DTA is probably large enough to beg and get special treatment, but I
don't actually want better treatment than others.
[2] And that's still a large legal gray area; e.g. can mozilla legally
sign add-ons of devs from countries with US sanctions/embargoes
[3]
https://blog.mozilla.org/addons/2016/11/23/add-ons-in-2017

We just released, and mozilla approved, a new maintenance release with the version number 3.0.7.

It fixes a few things, has a new disk writing backend to avoid low-memory issues in certain situations and adds Italian.

Release Notes

Another day, another release. We fixed a few things such as Save Form not working, and also, thanks to the awesome folks on babelzilla added es-ES, fr and zh-TW locales.

Get it from addons.mozilla.org and/or see our release page!

Nils
— for the DownThemAll! team

After a rather long long time, with many new things to address such as Australis, e10s, limited time we could spend on the project, a little lack of motivation after mozilla required mostly pointless signatures and announced to kill XUL add-ons…
And still spending lots and lots of hours, coding, improving and testing DownThemAll! in our free time and the awesome work by volunteer translators (on babelzilla) and fast review by Noitidart from the mozilla AMO team…

Here it finally is: DownThemAll! version 3
And an unavoidable fact is: nobody is perfect, so new versions means new bugs.
Therefore we make available the second bug fix release of DownThemAll!, with the version number 3.0.5.

Get it from addons.mozilla.org and/or see our release page!

Nils
— for the DownThemAll! team

PS: Also say hi to our new Git repository, currently hosted on GitHub.