Re: Emacs User Survey 2020 Results

[Jean Louis]

* Adrien Brochard <abrochard@gmx.com> [2020-12-10 00:31]:
> 
> > That is surprising result, maybe you remember I was expecting much
> > less people to respond. To me that speaks that there may be 7.3
> > millions Emacs users minimum, as I consider 1 survey submitted for
> > 1000 people who did not submit. This is a vague fact known in media
> > such as newspapers. It may not be.
> 
> That's interesting. Do you have any literature around that?

It comes from "Letters to editor" as single reader of a newspaper
writing such a letter was counted as 1000 other people. Cannot find
references. You could ask some professional newspapers how they regard
letters to editor. 

> My quick thinking on the topic is that Stackoverflow survey
> consistently reported 4.5% of developers using Emacs over the past
> years. Latest numbers indicate 4 million software engineers in the
> US, so that would be 180000 US software engineer Emacs users. Of
> course, that's rough given that there are Emacs users who do not fit
> that label. Very loose numbers say 21 millions software engineers
> worldwide, so by the same logic, that's 945000 users. I don't want
> to go too much into this discussion because really the data is
> missing, but it's interesting.

If they would say how many users did not answer the survey then you
could get some ratio to estimate better final numbers.

> > - your graphs are confusing and not common to me. It is not conclusive
> >    what you wish to present with the graphs such as "How do you use
> >    Emacs" where you are showing about 6000+ people using it for work
> >    and 2000+ people using it for studies. Your visual comparison is
> >    conflicting itself in my opinion as it does not make it conclusive
> >    if 2000 people among 6000 people use it for studies and for the work
> >    or only 2000 among 7300 use it for studies. As it is not definitely
> >    conclusive what you wanted to present I cannot be sure.
> 
> This question allowed multiple choice answers which makes graphing it
> always a bit tricky. I absolutely encourage you to look at the data and
> answer your question. As I stated at the top of the results page, this
> is a simple per-question analysis. I could have spent months looking at
> the data under every angle.

Programming is your helper. When you design it well from ground up you
never think again about it. I have been doing forms since
decades. Then I re-wrote the basic program 1-2 times in different
programming language.

There are good libraries available from Perl world. This page should
give you good insights to think about:

https://www.perlmonks.org/bare/?node_id=383250

Quotes:

,----
| Without knowing all the details, I'd say your best bet would be to
| validate the user input via JavaScript. Since JavaScript is
| client-side, you won't need to send the user input to the server and
| then back again if it's incorrect.
`----

where he gets the answer:

,----
| I'd say your best bet would be to validate the user input via JavaScript
| 
| Bad advice. Client side validation is fine, provided you accept that
| it is simply to improve the end user experience. It offers absolutely
| no security or reliable validation
| 
| Form validation is a server side task. Period. Do what you like on the
| client side, you still need to validate everything (again if using JS
| client side) on the server side.
`----

But that was 2004, today you have HTML5, you can validate pretty much
without Javascript on client side. Attempting to validate on client
side only would lead to insecurities. Using proprietary software even
more. 

> > - now the statistics "Can you list some of your favorite packages"
> >    where you have placed "other" as the longest item becomes less
> >    meaningfull because "Other" could be represented in words, such as
> >    that majority answered "Other" and then the rest you could display
> >    visually. That way the rest gets it visual meaning. This way, the
> >    longest item is so long that those named packages are visually not
> >    easily comparable to each other.
> > 
> > - same comment is valid for themes
> 
> Yes free text analysis on "long-tail" data is particularly difficult. I
> have mentioned it at the top of the results page.

Myself I do not use polls to analyze it for pure reason of analyzing
it. Rather to find out what is needed and wanted by majority and to
improve it. It can be a bridge that has to be repaired or school or
medical problem in some rural village. What starts cominng would would
be relevant. Analying it by percentages would not be useful for me.

If you say free form, then this is not really for analysis. Probably
it was not analyzed.

For accepting large number of users inputs I would use SQL
database. Those free form text can also be inserted into SQL
database. Its rigid data definition can equally well take care that
data too long cannot be inserted. Then comes the best part as
PostgreSQL at least supports full text relevance search.

For me it means that one could make one SQL query and find or group
relevant columns of data together. That would give the tabularized
summary.

Even better would be to have a survey running all the time where users
could just click to get the new fresh results be it tabularized or
graphical. 

> > - flycheck is not specifically error checking it is spell checking.
> 
> It is. https://www.flycheck.org/en/latest/
> You might confuse it for flyspell.

Thank you, did not know that.

> > - your Jypiter notebook can most probably be done also in Org
> >    mode. All the graphs could be also generated in Emacs as well and
> >    without proprietary external software. Graphviz and dot systems
> >    could be efficient.
> 
> It probably can but data analyst are more comfortable with Jypiter.

Does that information come from another survey?

Look at this query:
https://duckduckgo.com/?q=jupyter+vs+org+mode&atb=v154-3am&ia=web

Look at this result:
https://www.slant.co/versus/4411/15716/~emacs-org-mode_vs_jupyter

Rant:
https://rgoswami.me/posts/jupyter-orgmode/

Anyway, it does not matter. Personal preferenes. I would find it more
entertaining if all things were done with Emacs.

What we do with Emacs in Uganda nearby Bwindi Impenetrable Forest, we
track people's complaint, work, make police reports, we track stolen
goats, Org tables are used to calculate salaries and people look at
them and actually understand it. I am using Calc to calculate mineral
values. Common people watch and find it understandable. All videos and
images are converted, resized, managed by using Emacs as high level
language. All the 4000 pages of various websites are managed through
Emacs editing the PostgreSQL database entries. I know Perl, but if I
promote Perl I would recommend tools and stuff around and within Perl
environment. If Emacs, then I would recommend that within Emacs
environment. 

> > - from all the graphs that deserve to be the pie graph you have placed
> >    only one "how have you heard about survey" on the end.
> 
> That's actually not true. Pie graphs are good when the question is
> single choice answer. Most of the questions were multiple choice, which
> means that a pie graph would be confusing and the trick viewers into
> thinking that a user can only belong to one of the slices of the pie.
> Bar charts are not perfect, but they seem to reduce that risk. Maybe I
> could have also tried a bubble chart

That was one of possible options to understand it. But it is not
conclusive as there is no description on what you mean with
visualization.

Example:
https://emacssurvey.org/2020/how-would-you-characterize-your-use-of-emacs.png

It is now even less conclusive after your explanation if this means
one of following that 2500 people said "I use it for work" and also "I
use it for studies". As those doing study do not work yet. So the
visualization is missing the point to clearly tell the website visitor
what is the real result.

In that case graph alone is doing nothing but confusion. You would
rather say

- 65% of users said to use it for work
- 75% of users said to use it for studies

as those would be intersections of the union of results.

Example of more confusion:
https://emacssurvey.org/2020/for-how-many-years-have-you-been-using-emacs.png

Frequency? Frequency of what? Do you mean number of people? It is
interesting that some report using it over 40 years.

Another example:
https://emacssurvey.org/2020/which-os-do-you-primarily-use-emacs-on.png

Does that mean that 500 users use Windows and Mac and GNU/Linux
together how you explained it?

About 100 of them use other OS which they do not know which one it is,
Windows, Mac, GNU/Linux and BSD together. The logic is not consistent
with the explanation you gave.

Another example:
https://emacssurvey.org/2020/prior-to-using-emacs-what-was-your-primary-editor.png

Prior to using Emacs what was your primary editor?

- almost 1500 users answered NONE

- more than that answered OTHER which is is not consistent with your
  explanation, and not consistent with other 1500 users who used NONE
  editor before

- and not consistent to those 1500 users of vim, as it cannot be NONE
  and VIM and OTHER all together

About Org mode, that is also not how you explain, as than 1500 people
have said "I do not use Org mode" but among those 1500 people there
are those 1500 who "use it daily" -- sorry makes no sense.

Now all of those results are interesting and entertaining in the same
time. There may be some use of it. Comparing all the results to some
few bug reports from last weeks that have been pinpointed and handled
by Emacs developers, I still find the reports and their solutions so
much more remarkable and efficient. The survey itself, while
interesting, being conclusive or not conclusive, is not practically
useful.

And in regards to being representative, I would say you have got more
than representative data. With 1000 people would be already enough
to have it representative. You could and should say that you have
representative data. Congratulation.

But the visualization is wrong. It is not conclusive and is confusing
obviously.

There are 2 major things I see there missing and find the survey
wasted time and effort for nothing but few insights that could be also
by obtained faster and simpler way.

Those two things are: the objective or purpose and usability as one
possible purpose.

The objective for any survey, in my opinion, should be to improve
conditions. In this case to improve Emacs. But people were not asked
what they wish to improve. It seem that objective was just to make a
survey. That is why I say it is entertaining but less useful then
several solved bugs from last days on the bugs mailing list.

Usability I talk about is well explained here:
https://www.nngroup.com/articles/usability-101-introduction-to-usability/

If you read more from Nielsen you may also find that one need not have
more than 5 users to observe their habits and to find out what is
wrong and what can be improved.

Yet among 7300+ participants no question was answered in regards on
what they wish to improve in Emacs, what is their major problem, or
where usability could be improved.

I do hope you will make a checklist of those items you find yourself
reasonable and that it gives better more usable results in future.

Jean

P.S. You said before the surve you would give the unchanged raw data
back to public. I have signed my information with PGP key and sent it
to you. Now I cannot find my PGP information "raw" and for that reason
myself I do not find it more than funny time, nothing serious. When
you say to publish it all, then why not really do it.