cloc (http://cloc.sourceforge.net/#Languages) supports more languages
than sloccount (e.g. Scala, Javascript), and could be used to count
the lines of code in every package.

This has to be carefully tested, since it needs to run of very big
packages, and be fast enough.

Cheers,
--
Matthieu

Heya,

I started looking at this since cloc would also be able to provide a
matching between files and programming language.

I tested cloc directly on the testdata folder and here are some results:

time sloccount --addlangall testdata/
real    0m10.920s
user    0m2.768s
sys     0m2.656s

time cloc testdata/
real    0m34.636s
user    0m16.188s
sys     0m6.612s

time cloc --by-file-by-lang -csv testdata/
real    0m35.695s
user    0m16.764s
sys     0m6.788s

The last one provides an output matching each file with a programming
language.

Besides the time difference one needs to also notice the difference in
results. Cloc has many more languages but also categorizes sourcecode
differently. I.e instead of ansic it has C and C headers.
It also splits results into source code, blank lines and comments. I
haven't investigated what sloccount does for that matter.

On a plus side cloc can output json or csv so parsing the output would
be easy.

Cheers,

Orestis

Heya,

So here are also some benchmarks, running sloccount and cloc on linux
and chromium.
(I had to run latest cloc from github because the one in testing has an
older release that affects chromium :D )

- time sloccount --addlangall linux-4.4.6

Totals grouped by language (dominant language first):
ansic:     13550489 (96.81%)
asm:         283173 (2.02%)
xml:          49474 (0.35%)
perl:         46694 (0.33%)
makefile:     33142 (0.24%)
sh:           10942 (0.08%)
python:        9705 (0.07%)
cpp:           5289 (0.04%)
yacc:          4357 (0.03%)
lex:           2340 (0.02%)
awk:           1128 (0.01%)
pascal:         231 (0.00%)
lisp:           218 (0.00%)

19.34s user
16.49s system
49% cpu
1:12.70 total


- time cloc-1.66.pl --by-file-by-lang -csv linux-4.4.6

files,language,blank,comment,code
22558,C,2217923,2064183,11330129
17092,C/C++ Header,416288,713217,2673966
1412,Assembly,47490,110738,242086
175,XML,3419,242,49467
202,Perl,9200,7412,46189
2105,make,8945,8950,41290
173,Bourne Shell,1769,3161,9650
55,Python,1765,1532,9606
8,yacc,649,355,4357
8,lex,292,289,1823
51,Bourne Again Shell,338,261,1580
1,C++,231,58,1579
10,awk,132,129,1130
3,NAnt script,130,0,462
2,HTML,58,0,378
3,Pascal,49,0,231
1,Lisp,63,0,218
1,Objective C++,55,0,189
1,m4,15,1,95
6,XSLT,13,27,71
1,vim script,3,12,27
1,Windows Module Definition,0,0,8

159.95s user
21.10s system
81% cpu
3:40.93 total

- sloccount --addlangall chromium-browser-49.0.2623.108

Totals grouped by language (dominant language first):
cpp:       11192924 (61.48%)
ansic:      3524388 (19.36%)
xml:        1209320 (6.64%)
python:     1063920 (5.84%)
asm:         605129 (3.32%)
java:        368846 (2.03%)
sh:           89957 (0.49%)
perl:         73187 (0.40%)
makefile:     31024 (0.17%)
objc:         14698 (0.08%)
tcl:          11315 (0.06%)
yacc:          9847 (0.05%)
sql:           2765 (0.02%)
cs:            2696 (0.01%)
lex:           2259 (0.01%)
pascal:        1288 (0.01%)
lisp:          1247 (0.01%)
awk:            497 (0.00%)
ruby:           141 (0.00%)
sed:             37 (0.00%)
php:             15 (0.00%)
exp:             11 (0.00%)

38.30s user
51.66s system
48% cpu
3:04.32 total


- time cloc-1.66.pl --by-file-by-lang -csv chromium-browser-49.0.2623.108

files,language,blank,comment,code
38879,C++,1557112,1181535,9426443
7805,C,470331,567084,2890280
34270,C/C++ Header,762796,1223172,2851525
891,XML,23141,9541,1208785
3607,JavaScript,138882,305771,1044159
7693,Python,282771,400973,1027855
7577,HTML,121950,56773,973780
1878,JSON,1660,0,931914
790,Assembly,46770,43471,628908
2789,Java,71969,112496,368684
1620,Objective C++,51254,40610,237545
630,Bourne Shell,28007,29811,166525
1338,IDL,9829,0,79835
854,CSS,14833,7664,75294
131,Perl,11888,12271,73313
91,MSBuild script,0,35,59338
102,m4,5389,946,49070
371,make,5047,5361,24735
33,Go,2316,1788,15386
46,Tcl/Tk,1393,2597,11335
240,Protocol Buffers,4128,9532,11180
113,Objective C,2588,2162,10431
177,CMake,1798,1215,8558
7,yacc,1170,955,8068
45,Windows Module Definition,82,126,5960
24,XSLT,895,1191,4092
106,Bourne Again Shell,776,1453,3987
60,DOS Batch,519,467,2894
97,SQL,111,171,2740
35,C#,474,1079,2677
29,Lua,458,259,2386
45,Windows Resource File,553,1200,2078
4,XSD,184,1133,1898
19,diff,73,400,1745
52,YAML,256,332,1724
9,Cython,459,220,1657
4,lex,262,116,1394
16,MATLAB,218,188,1031
10,Lisp,147,165,1025
21,XHTML,13,14,944
10,HLSL,161,138,690
38,Jam,231,513,682
8,Expect,21,87,652
12,Ant,114,290,635
8,Pascal,167,1575,602
1,WiX source,80,59,515
8,awk,70,131,503
14,Ruby,59,81,309
8,vim script,75,112,264
2,Racket,17,51,255
4,DTD,44,166,245
1,Korn Shell,39,46,223
4,XAML,30,50,90
4,Standard ML,9,0,61
4,Groovy,17,96,57
4,PHP,9,0,54
9,Verilog-SystemVerilog,0,0,52
4,sed,12,21,38
1,R,5,5,37
1,Prolog,1,-1,32
1,Swift,2,0,7

297.78s user
100.08s system
69% cpu
9:34.93 total

Hey,

That point makes me feel pretty confident that cloc can be broken for
other packages. It would be relevant to carefully log errors when we
run the plugin on the whole archive!

That is awesome to have the comments, it adds a good value IMO. Can't
wait to query debsources, asking which license is the most popular
among non-commented code, and other funny queries :)

That's significantly slower than sloccount, but well, looks ok for
something as big as the linux package. Have you had the chance to see
if it parallelizes well?

... is that even true? That's off-topic now, but I'm looking forward
to see in which files of chromium we can find php and ML :)

Thanks for you work on this bug!
Cheers,
--
Matthieu

Hi all,

I noticed that in the "package info" box of sources.debian.org, the "SLOC" field
doesn't recognize Rust. Example [1].

After doing some research I believe it's because of the somewhat outdated
sloccount program. Similar report can be found here regarding Scala [2].

Do we have any plan of updating this infrastructure? In my use cases, I turn to
sources.debian.org and that SLOC field quite often if the package is not
maintained on Salsa.

Thanks,
Wentao

[1] https://sources.debian.org/src/rust-bat/0.24.0-1/
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=736486

Hi,

Indeed, unfortunately sloccount doesn't seem to be actively
maintained, a merge request for Rust support didn't get any attention:
https://sourceforge.net/p/sloccount/code/merge-requests/5/

I'd love to see support for more languages. We'd need to:

- Find a suitable alternative (supports a superset of sloccount
  languages, has decent performances).

- Replace the execution and parsing of sloccount output by this
  alternative (in
https://salsa.debian.org/qa/debsources/-/blob/master/lib/debsources/plugins/hook_sloccount.py?ref_type=heads).

- To keep things consistent, run it on the whole archive to replace
  sloccount results.

Insights on the first point are particularly welcome.

Cheers,
--
Matthieu