visidata downloads stuff from https://visidata.org/ behind my back.
Please don't do that.

Traceback (from a crash provoked by syscall tampering[*]):

   File "/usr/lib/python3/dist-packages/visidata/urlcache.py", line 25, in urlcache
     with urlopen(req) as fp:
   File "/usr/lib/python3/dist-packages/visidata/plugins.py", line 68, in reload
     self.source = urlcache(options.plugins_url or vd.fail(), days=0)  # for VisiDataMetaSheet.reload()
   File "/usr/lib/python3/dist-packages/visidata/threads.py", line 235, in _func
     func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/visidata/threads.py", line 215, in _toplevelTryFunc
     t.status = func(*args, **kwargs)
   File "/usr/lib/python3.9/threading.py", line 910, in run
     self._target(*self._args, **self._kwargs)
   File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
     self.run()
   File "/usr/lib/python3.9/threading.py", line 930, in _bootstrap
     self._bootstrap_inner()


[*] strace -f -o /dev/null --inject=connect:signal=SIGABRT visidata /etc/fstab

Hi Jakub!

- VisiData fetching the list of startup messages he first time each day
that VisiData is used is documented in the privacy policy:
https://www.visidata.org/privacy/.
-
- As noted in the privacy policy, the network request can be turned off by
adding `options.motd_url=None` to your `~/.visidatarc`.
There is definitely a bug in v2.2.1 and earlier, where VisiData will access
the website each time it is used, in order to grab the list of plugins.
This bug was fixed in v2.3:
https://github.com/saulpw/visidata/blob/stable/CHANGELOG.md#bugfixes-5. I
have not updated the VisiData on debian yet post v2.2.1, because I wanted
to be certain of a stable candidate on debian. v2.8 coming up will have
that fix, and is a candidate for updating the Debian package. I can leave
this bug open as unresolved until that is done.

For now, you can set `options.plugins_url=None` to your `~/.visidatarc` to
turn that off. `open-plugins` will not work, but if you do not want your
software making network requests, you probably would not want that feature.

For a detailed explanation on why the author chose to have the network
requests on by default, please see this discussion:
https://github.com/saulpw/visidata/discussions/940

Short version:

  • This works: printf 'options.plugins_url=""\noptions.motd_url=""\n' > ~/.visidatarc
  • Your earlier suggestion does not work anymore.


Boring technical details follow.

I am a nosy bystander.
I do not use visicalc (I use awk and sqlite 😉).

By default "visidata" causes two files to be created (by downloading).

    ~/.visidata/cache/https%3A%2F%2Fvisidata.org%2Fmotd-2.2.1
    ~/.visidata/cache/https%3A%2F%2Fvisidata.org%2Fplugins%2Fplugins.jsonl

One of them "leaks" the visidata version number.

https://sources.debian.org/src/visidata/2.2.1-1/visidata/motd.py/?hl=8#L16

Your suggested option did NOT have any effect (both files are created):

    echo options.plugins_url=None > ~/.visidatarc

However, this DID work (neither file is created):

    printf 'options.plugins_url=""\noptions.motd_url=""\n' > ~/.visidatarc

A full transcript is attached.

This bug still exists in the version of visidata released with debian bookworm.

Debian users do not expect software provided through debian to be phoning home for reasons unrelated to the function of the software. In my case, visidata was phoning home while working on GDPR data...

Normally, package maintainers in debian patch out this functionality before releasing the software to users. Please do so.

I hope it's not relevant here that the package maintainer of visidata in debian is also the upstream author, and this phoning home provides upstream with some sort of revenue stream, and this is why this bug hasn't been dealt with yet...

Hey stqmts+5ehe03cek0yy8,
debian is also the upstream author, and this phoning home provides upstream
with some sort of revenue stream, and this is why this bug hasn't been
dealt with yet...

The creator of VisiData goes into great depth here about his decision to
enable motd by default: https://github.com/saulpw/visidata/issues/913

The privacy page is here: https://www.visidata.org/privacy/.

In particular: VisiData fetches the list of startup messages with a single
HTTP GET. The first time each day that VisiData is used, it downloads a
single small file. This access is also logged by the website.
This network request can be turned off by adding options.motd_url=None to
your ~/.visidatarc.

This daily count of users is what keeps us working on the project, because
otherwise we have feel like we are coding into a void. There are some
extremely dedicated users who talk to us about how much they love the
project, and engage with us, but by-and-large we only found out people were
using it when they were unhappy with something. Knowing that there is a
steady increase of people using it, motivates me to fix bugs in my evenings
after a full day at my day-job.
the code, it is all open source:
https://github.com/saulpw/visidata/blob/develop/visidata/motd.py. It just
does an HTTP Get. The process to get a package accepted into Debian was
brutal. The user @geekscrapy is extremely privacy conscious, and he did
tell us that he made the choice to turn motd off, but otherwise has not
voiced to us any concerns. He hangs out in the VisiData discord, if you
wanted to chat any further about your concern: bluebird.sh/chat. Also, in
case this matters to you, we wrote about any occurrences of swap files
here: https://github.com/saulpw/visidata/issues/140.

I read that you are also concerned about any behind the scenes revenue.
VisiData has no revenue stream.

Saul, as a creator, has a github sponsors:
https://github.com/sponsors/saulpw and a patreon:
https://www.patreon.com/saulpw/posts. People sponsor if they want to
support his work, and no other reason. Examples of something that patreon
money has supported is paying a web developer to build a better-looking
website for VisiData.

the last octet of visitors IPv4 addresses or the last twelve octets for
IPv6 addresses. This still allows useful logging for statistics, but
makes identification of single users pretty hard.
TTBOMK, this is what is considered GDPR compliant, but IANAL and
IANADPO.

If there were a global configuration file, such as /etc/visidatarc,
setting the motd_url could be done also as part of the Debian package
installation using "debconf".

Maybe at some point, you could even add a startup wizard/dialog:
"Download motd files? [Always] [Yes] [No] [Never]"

Note, that your statistics usage.tsv seem to end on 2023-04-09. Did I
miss the apocalypse?

Btw. you can get a statistics about installed visidata packages here:
https://qa.debian.org/popcon.php?package=visidata
popcon (popularity contest) is opt-in, i.e. the real number is higher.

the last octet of visitors IPv4 addresses or the last twelve octets for
IPv6 addresses. This still allows useful logging for statistics, but
makes identification of single users pretty hard.
TTBOMK, this is what is considered GDPR compliant, but IANAL and
IANADPO.

If there were a global configuration file, such as /etc/visidatarc,
setting the motd_url could be done also as part of the Debian package
installation using "debconf".

Maybe at some point, you could even add a startup wizard/dialog:
"Download motd files? [Always] [Yes] [No] [Never]"

Note, that your statistics usage.tsv seem to end on 2023-04-09. Did I
miss the apocalypse?

Btw. you can get a statistics about installed visidata packages here:
https://qa.debian.org/popcon.php?package=visidata
popcon (popularity contest) is opt-in, i.e. the real number is higher.

Re: Anja

This privacy policy is incomplete. You are collecting PII (the IP
address of users of the program, not just the website visitors), but
there is no data protection officer defined, no contact address for
enquiries, etc. As you probably have users in Europe, the GDPR apply,
and this is a violation.

Collecting PII needs a legitimate reason, and the one stated here "we
like to have numbers of users" sounds nice to have, but I really don't
see it outweighing the interest of the user to remain private.

TBH, you are on very slippery ground here and would be far better off
by turning this off for legal reasons.

As already mentioned, there is the Debian popularity contest (which is
opt-in), and it does have visidata numbers:

https://qa.debian.org/popcon.php?package=visidata

This is how this works in Debian. Please you can keep the feature, but
make it opt-in.

There is no written policy for this yet, but every other package I
know with such a feature has turned it off in the packaging.

Thanks.

Christoph

I am submitting a patch for the version of visidata currently packaged in
debian sid to eliminate the default behavior of the application phoning home,
which has raised some privacy concerns.

When this patch is applied, visidata will not phone home by default anymore.
Users retain the possibility to enable the phoning home functionality by
explicitly specifying the options.motd_url option in their visidatarc
configuration file with a suitable value.

There is an existing lintian tag, privacy-breach-generic, which can be used as
an indication that phoning home is considered undesirable behavior within the
debian project. Additionally, other packages in debian, for example gpsbabel
[1], have also been modified to remove similar privacy issues, further
establishing a precedent for this change.

While this behavior is not explicitly codified in debian policy yet, phoning
home goes against user expectations when utilizing debian-provided packages.

[1] https://salsa.debian.org/debian-gps-team/pkg-gpsbabel/-/tree/master/debian/patches?ref_type=heads

Hi Christoph,

Thanks for pointing out GDPR-specific compliance measures that VisiData is
missing. I began looking into it, and I'm aiming to update our privacy
policy this week.

My understanding is that "legitimate interests" requires a straightforward
opt-out mechanism (which VisiData does provide), but doesn't require
opt-in. We're going to ensure we meet legitimate interests compliance.

My plan is:
* to add language related to "legitimate interests" as the GDPR legal basis
* automate the deletion of individual data the day after the daily usage
counts are calculated
* create a process for data access and deletion requests if a person wants
their data deleted earlier

Again, thank you for drawing our attention to this!

Best,
Anja

Re: Anja

The GDPR mandates privacy by default, so opt-in unless you have good
reasons otherwise.

But whatever the GDPR says, Debian wants no software to call home
unless it's for really good reasons. Please consider following project
best practices.

Christoph