SonarQube

I'm interested in tools for software correctness; that's why I've tried Hypothesis on my current side project; written models in Alloy; run libFuzzer on Tintri file system components; and done other investigation.

When I was introduced to SonarQube it was along the lines of "oh, they do a software service like that" so I tried them out on Soffit. You can see the dashboard here: https://sonarcloud.io/dashboard?id=mgritter_soffit

I was extremely disappointed with the experience.

Initial Setup

I've got a public Github repository, so I thought it would just be "point the tool at the repository" and it would run a source scanner or perform some other tests.

SonarCloud does have a way to access Github, but it's listed as "Beta" and it wanted to install a GitHub app; I wasn't sure what I was signing up for. So I clicked over to the other tab.

What's a key? Why do I need one? Is it a password? (It's not a password.) What decade are we in, that no contextual help is available?

OK, so I typed some stuff in and then next page wants me to download their tool and run some stuff. I didn't think that's what I was getting, so I figured out how to delete the project and successfully hooked the SonarCloud project up to Github.

Running the probe

After all that, I was still at the same place: "download the scanner for Linux, install it, and then run this command." This is the very opposite of the SaaS offering I was expecting. I just want to find bugs in my code, and now you've got access to my code! (Actually, the GitHub app they installed didn't have permission to read code, which I found suspicious.)

OK, I'll play around a bit longer. It's a 70MB installer so it takes a bit to download. I unzip it to install and it looks like a Java app, naturally bundled with the JRE. I follow the directions and it churns away for a couple minutes:

mark@ubuntu:~/soffit$ sonar-scanner \
>   -Dsonar.projectKey=mgritter_soffit \
>   -Dsonar.organization=mgritter-github \
>   -Dsonar.sources=. \
>   -Dsonar.host.url=https://sonarcloud.io \
>   -Dsonar.login=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
INFO: Scanner configuration file: /home/mark/sonar-scanner-3.2.0.1227-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.2.0.1227
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 4.4.0-89-generic amd64
INFO: User cache: /home/mark/.sonar/cache
INFO: SonarQube server 7.5.0
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=832ms
INFO: Server id: BD367519-AWHW8ct9-T_TB3XqouNu
INFO: User cache: /home/mark/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=179ms
INFO: Load/download plugins (done) | time=107051ms
INFO: Loaded core extensions: branch-scanner
INFO: Process project properties
INFO: Load project branches
INFO: Load project branches (done) | time=158ms
INFO: Load project pull requests
INFO: Load project pull requests (done) | time=151ms
INFO: Load branch configuration
INFO: Load branch configuration (done) | time=8ms
INFO: Load project repositories
INFO: Load project repositories (done) | time=195ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=176ms
INFO: Load active rules
INFO: Load active rules (done) | time=4427ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=159ms
INFO: Project key: mgritter_soffit
INFO: Project base dir: /home/mark/soffit
INFO: Organization key: mgritter-github
INFO: -------------  Scan mgritter_soffit
INFO: Base dir: /home/mark/soffit
INFO: Working dir: /home/mark/soffit/.scannerwork
INFO: Source paths: .
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Load server rules
INFO: Load server rules (done) | time=1936ms
WARN: Property 'sonar.abap.file.suffixes' is not declared as multi-values/property set but was read using 'getStringArray' method. The SonarQube plugin declaring this property should be updated.
INFO: Index files
INFO: 110 files indexed
INFO: Quality profile for py: Sonar way
INFO: Quality profile for web: Sonar way
INFO: Sensor SonarJavaXmlFileSensor [java]
INFO: Sensor SonarJavaXmlFileSensor [java] (done) | time=2ms
INFO: Sensor HTML [web]
INFO: Sensor HTML [web] (done) | time=622ms
INFO: Sensor Python Squid Sensor [python]
WARN: Metric 'comment_lines_data' is deprecated. Provided value is ignored.
INFO: Sensor Python Squid Sensor [python] (done) | time=1232ms
INFO: Sensor PythonXUnitSensor [python]
INFO: Sensor PythonXUnitSensor [python] (done) | time=5ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=3ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=49ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Reading UCFGs from: /home/mark/soffit/.scannerwork/ucfg2/java
INFO: UCFGs: 0, excluded: 0, source entrypoints: 0
INFO: No UCFGs have been included for analysis.
INFO: Sensor JavaSecuritySensor [security] (done) | time=4ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading UCFGs from: /home/mark/soffit/ucfg_cs2
INFO: UCFGs: 0, excluded: 0, source entrypoints: 0
INFO: No UCFGs have been included for analysis.
INFO: Sensor CSharpSecuritySensor [security] (done) | time=1ms
INFO: SCM provider for this project is: git
INFO: 17 files to be analyzed
INFO: 15/17 files analyzed
WARN: Missing blame information for the following files:
WARN:   * start.py
WARN:   * test.html
WARN: This may lead to missing/broken features in SonarQube
INFO: 2 files had no CPD blocks
INFO: Calculating CPD for 17 files
INFO: CPD calculation finished
INFO: Analysis report generated in 217ms, dir size=651 KB
INFO: Analysis reports compressed in 50ms, zip size=177 KB
INFO: Analysis report uploaded in 1041ms
INFO: ANALYSIS SUCCESSFUL, you can browse https://sonarcloud.io/dashboard?id=mgritter_soffit
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at https://sonarcloud.io/api/ce/task?id=AWckdA9ARuF_QXgqNlw7
INFO: Task total time: 14.351 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 2:18.386s
INFO: Final Memory: 26M/176M
INFO: ------------------------------------------------------------------------

Should I be reading all of this? Do I care? OK, back to the web portal.

Bugs detected

The dashboard says it has two bugs! Exciting! Except...

It's complaining about missing HTML elements in a test output file. A test output file, I should mention, that was not checked in. OK, so, yeah, if this were production code, it would matter. But I kind of expected the tool to look at what was actually in the repository, not just whatever files it happened to find in the directory.

Code smell

OK, 324 "code smell" issues. (That means, not a bug, but a sign something else is wrong.) So I open it up to this undifferentiated mess:

SonarQube doesn't like my naming convention. It didn't highlight things that were outliers from the convention I had been using. No, it brought its own standard to the table and dinged me hundreds of times for not following it.

It also raised issues for all the FIXMEs and commented-out code.

What about the "critical" issues, maybe those are more relevant?

Well... OK. Maybe lead with that next time? But those are hardly critical to fix.

I looked for a way to disable the naming convention check, or alter it. SonarQube has different "quality profiles" but I didn't seem to have any other existing option and I couldn't figure out how to create a new one. I could bulk-close the issues but they would probably just be reported again later?

Tests and Test coverage

I have unit tests! Did it pick up that I had them?

It looks like the scanner (or its Python plugin) does not run coverage or even unit test execution by itself, but it can import a result from other tools: https://docs.sonarqube.org/display/PLUG/Python+Coverage+Results+Import

For unit tests, it says it works with Nose, which I'd never heard of before:

Conclusion

This was a waste of time. SonarQube basically ran a linter, but pylint would have been easier to run and configure. It didn't really help me focus on what was important by breaking "code smell" down by category, so I could see what types of issues were being found.

SonarCloud didn't run the linter on the source repository I pointed it at, so its "GitHub integration" is oriented around other features than source control. (Like change tracking, I guess? Except it doesn't show any of my commits?)

SonarQube doesn't know how to run my unit tests and it doesn't provide any comprehensive guidance for how to do so from the web portal, so I'm not motivated to use some other package to try to get it working.

The SaaS offering may be great if you've already got a CI/CD pipeline and want to start extracting data from it, compared with running your own server. But the experience of trying out the offering on an existing open-source project is very poor, and its failure to identify any genuine bugs means I'm not interested in making the investment.

Perhaps the Java experience is better; at work we've used FindBugs for Java, which does a good job of immediately finding actionable problems, and prioritizing them in its reporting.

The postings on this site are my own and do not represent DataDirect Network's positions, strategies, or opinions.